It becomes tenuous if one starts randomly changing laws (of physics, especially) then there needs to be loads of special-cases to cover up.
Magic is an easy short-cut, and avoids the need of working out all the annoying details up front. But it also means that, at any time, magic can change the course of the game in a completely unpredictable way.