Adding to the issue (and i'm not sure if it's related or not) but casting Mirror Image freezes the game for me. This in addition to the behavior described you you fine gentlemen above, is currently affecting my system as well.
OS: WIndoes 7 x64
Processor: AMD Phenom II x4 975 ~3.6 GHz
Video Card: AMD Radeon HD 6900 series (latest drivers)
DirectX: 11
RAM: 12 GB.
My guess is these issues are all related to character avatar ( and i think someone might have mentioned this earlier) and caching data. I really hope we see a workaround of fix for this memory issue, since it's making the game quite sluggish and tedious to play.