Well for starters you know the problem isn't with ESO, but this is a router or modem setting issue. You might want to check the ISP's forums for any similar issues/solutions. Personally I'd check for network congestion though.

-Make sure the router/modem does not respond to WAN pings. Might cause ESO to be put on hold(disconnect), as the router/modem is already busy responding to WAN signals.
-Make sure both the modem/router discards any unwanted, double or errored packages/signals.
-Make sure the router/modem does not bounce back any packages anywhere, but discards them instead.
-Make sure the router/modem does not recheck/remake the connection every few minutes.
-Make sure your router/modem aren't overheating, serious gaming can stress out regular equipment. MMO's are serious gaming, due to the constant commands being sent to the gameserver. This happening is doubtfull though.
-Make sure the router/modem does not detect certain signals as a reason to kill the connection. This could happen if one of the ESO's commands happens to be the same/similar to the router/modem's kill connection command.

Other than that, walk through every possible setting on both the modem and the router. To check if any of them could be causing a disconnect somehow. Hope this helps.

PS: I doubt your issue is a port issue, as you can already play.