Your wizard does self buffing so he/she isn't using pure physical power to lift tree trunks
But riddle me this, Batman: How much physical power was he using to lift tree trunks? You can reasonably say "I don't care how much, I just care what he's doing and not how he's doing it," but you can't reasonably tell anyone the answer to that question, because there isn't one. If this were a math problem on a test, the answer would be "Not enough information."
Might as well combine Intellect and Might into a single stat called Capability, then just say "Maybe the character is exerting enough power on the wall to destroy it in its current state, or maybe he's using elaborate scientific planning to dissolve the mortar between the bricks using his own concoction of acid by MacGuyvering readily available substances together... it doesn't matter, because he's getting past the obstacle, because he's capable!"
"What is he capable of, specifically?"
"... ACCOMPLISHING THINGS! GYAH! WHY DO PEOPLE NOT UNDERSTAND THIS STAT?!"
Now, if you'd excuse me, I'm going to go slay some dragons with my high Defeatery stat.
At best, abilities are always going to be a simplified abstraction that is useful for creating a coarse simulation. They are a compromise and a simplification of what is in actuality a highly complex set of bodily and mental capabilities. It is easy to find fault with them because they will never simulate the exact physical universe. But that is true of games in general -- they can only provide a rough verisimilitude of the real thing. In the end then, it comes down to what the designer wants to portray. Abilities only provide basic building blocks, an edifice upon which can be erected a slightly more refined simulation provided via features such as skills and feats.