Gotta leave and come back later. It's all you can do. Also, the 1st time I tried the portal without a pet, it worked. No idea if that matters or if I just got lucky.
Seems to have been a problem with ether the pet or the distance from portal when cleared. I was right in front of Haskill's table without my pet the second time.