I am rubbish at PVP and dueling but I do like to watch others duel at Bergama Wayshrine (which is where everyone seems to go on PC EU).
It would be nice if ZOS fixed up Hammerdeath Arena and made it more enticing to duelists. Have matches. Have prizes for the winners. Have guild traders there and unique merchants that sell unique items you can't get elsewhere. Put a wayshrine right at the entrance. This would all help drum up passing interest/trade and make it more likely that players visited Hammerdeath Arena, either to watch or participate.
I would say allow players watching to place bets on their favorite duelist, but I don't think that would be allowed, but the duelists themselves could at least participate in matches to win prizes, like an Olympic Games of dueling. Create a leaderboard, facilitate a queuing system (much like IC or Cyrodiil) where if you are waiting for your match, you can go do other stuff and port to the Arena when it's your turn to duel.
Add in daily dueling quests or the like to further entice.
It is such a wasted space.