Maybe they're trying to tell you something. Mine does the opposite. I arbitrarily switch to first person whenever I go in to my inventory. I could just be hitting the wrong button though. Have to check.
I play mainly in 1st person, especially when walking through doors - I never experienced such a switch - I am on PC though, maybe it is a console issue, what you experience.
Edit: but I experience other unintentional switches in combat, when I am using a bow - it sometimes switches me to 3rd person with the second shot in a row.