I've had that issue with gamepads since analog sticks existed. It's almost always caused by the same thing.

Joysticks have a center location and wear out over time so they don't come back to fully center, often causing camera/movement creep.
A joystick that is registering movement when a game is loaded, because it isn't back to center, is often considered at center at that point so that it no longer responds to that directional command in the game.

Workarounds:
1) Do not hold the left stick in any way when in a loading screen or especially when starting up the game because this will likely cause this.
2) Only way to fix a worn out stick is a controller replacement, unless you somehow know how to open yours up and repair the mechanism without breaking it or something else.

I don't buy used controller for this reason, most of them have a worn out stick with "creep" and issues registering starting the game because of it.

Your issue could be something else, but this is an idea. You could try another gamepad to verify, which would be standard troubleshooting to eliminate the easy to test variables.