I had that too. This is how i fixed it. Make certain that the chosen resolution fits the chosen window size, eg. I have a 1920x1200 monitor and in full screen mode the game would show that resolution. After switching to window mode I had to set the resolution to less than that (1600 something) for it to switch successfully.