Yes, that does matter. You need to use matching bait, i.e. worms in saltwater/guts in lakes, or you get a lot more used bait results.
Additionally, the Artaeum Pickled Fish Bowl food makes a big difference in how many rare fish you catch, so it's a good investment if you are going for Master Angler.