I agree with the lack of swoop upgrades and bad inventory layout as mentioned above.
One thing that also annoyed me is how the game pushes you to make a Soldier character, since the only real difference between the classes is that the Soldier wins fights more easily. I first played the game as a Scoundrel and had to die and reload millions of times until I finished the game, then played as a Soldier and honestly didn't die once.
There should have been more differences in the way the game moves on depending on the class you choose (class-related quests, perhaps? I loved in games like Quest for Glory how thieves could do Thieves Guild quests, pick pockets, break into mansions etc, for example) and in some cases alternatives to combat for non-Soldier characters.
I was also annoyed by the small number of possible endings. A Light side ending and a Dark side ending, nothing more.