Gonna need more specifics. What abilities, what level, how many mobs you taking on, what add-ons installed, etc? It's broke help me ain't gonna cut it.
Are you stunned?
Other than that I think it's a known issue of sort, a quick double tap of ESC will bring them back.
I am thinking this is more of a lag related issue. Sometimes it takes the game a solid couple seconds to register a change in target. Even if you clearly see the outline on your screen. To you it looks like you are targeting that pirate, but to the game it looks like you are still targeting that lizard.
Something similar happens to me often in game. I highlight one target only to change my mind, and highlight another, because I decide the second target is what poses the greatest risk. Even though on my screen the second target is firmly locked in. My first attack will still hit the first target I had highlighted just a couple seconds prior.
I can't really speak to targeting non hostile creatures. At some point I learned to avoid hitting them unconsciously. I guess I learned to compensate for all of the wasted damage. By placing my reticule high, and always in center mass. My character is also tall so that may be helping somewhat. Anyway try aiming high, and see if that helps clear things up.