Though my previous comment hasn't been approved yet, I tested this issue with a fresh single player game, and it worked flawlessly. No stuttering, no lag, nothing. So as OP suggests, it's likely something such as ants compiling food which eventually just gets to the point where the game can't handle to load of items.