I honestly think it's a throttling issue. Spell effects and lighting are sending temps through the roof. The reason you're getting better performance after restarting the game might be due to heat dissipation while the client isn't running. After 5 mins or so I bet you'll see performance nosedive again.
Cast a bunch of spell effects, pause the game and wait a while, then check temps. It's like the shaders are completely overloaded due to the backgrounds and the complexity of the effects, so even during a pause they're running in overdrive constantly. Character models just standing around has performance at normal, yet when a lot of effects are on-screen the performance gets trashed.
For what it's worth, my performance increased somewhat by limiting my framerate so that when a lot of effects aren't on-screen my game isn't running all-out, then when a lot of effects hit the screen I still have some room to budge until temperature throttling kicks in when temps increase.
There was another post on the Steam community site about how this game heats up the GPU so much even on high-end systems. It would be very advantageous for people to check temps during the framerate drops since I'm fairly sure the performance problem is due to temperature throttling -- and if it is, Obsidian needs to find ways to tone down shader effects so they don't burn out their customers' computers.