I changed to my 4G modem and it works. Strangest thing ever.
EDIT: And yes, I tried resetting my cable modem, restarting PS4 and everything. The connection was NAT 2 and everything should be working just fine; NA server was working just fine. For some reason I couldn't connect from that IP address to ESO server.