Buy bags from a bag vendor, train that stat on the mount training skill

I have a video guide explaining it, but essentially what has been said above is correct. Purchasing pack space from a pack merchant will increase your inventory space by 10, while training in the capacity perk at the stables will increase your carrying capacity by 1 (at all times, not just when riding a mount). Bank space can also be increased, although this will not effect your actually carrying capacity while you're out adventuring in the world (but will make it easier just to stop by town and throw everything in the bank so you can head back out quick)

Here's a link to my post with the video guide - at the end there is a spreadsheet shown that actually shows the costs of each upgrade.

http://forums.elderscrollsonline.com/en/discussion/comment/1846061/#Comment_1846061