Same issue for me - about 30 hours in, and the save bloat is getting a little out of hand. I don't mind 20-30 second load times in areas I'll be spending a decent amount of time in exploring and talking, but for screens I'll be spending 2 seconds on, it's actually daunting enough that I don't want to play.
Edit: After creating two new characters, I can confirm that the loading time is quite a bit faster (only 10-12 seconds instead), though still slower than I remember it being upon first starting to play.