Today, I just clicked a tiny bit outside of a patron by accident and instead of doing nothing, it ended my turn (without the hourglass being highlighted). This is on the PS5 client. Not sure if the end of turn area is too large on other platforms too. It should only end the turn if the cursor is actually on the hourglass, not outside of it.