It depends on the dungeon. If you're in a public dungeon, there will be a loot timer. If you kill too many in a row you will not get loot for a bit. I think there is a 5 minute timer that starts when you loot the first one then another 5 minute timer on the second one. I may be off a bit and it may vary from dungeon to dungeon.

If you're talking about a group dungeon, some named monsters will not count as bosses. Any boss that drops loot should drop loot on every run. Any named monster that doesn't drop set loot will never drop set loot (unless it's the same set loot that can drop randomly from any mob)