Mouse double clicking and panning my screen?

So every once in a while if I am in my inventory or in combat single clicks will result in double clicks resulting in something I do not want.  Also in combat if I select a spell sometimes it will double click and then unselect the spell and pan the camera.  I have googled this issue and it seems like some people have it but no solution...maybe I missed it?  This doesn't happen THAT often but enough to annoy me lol!