Because when you're designing a game, you have to pick the core features you want your game to focus on--you don't have the time nor the money to do everything.
You mentioned BioWare not having used a Z-Axis since MDK2 in their games. As you well know, BioWare's focus is always on their detailed characters/NPCs and the overall narrative. If having a Z-Axis doesn't strengthen their core focus in any way, why would they feel the need to add it? Because other games, that aren't even RPGs (and thus have different areas of focus) do so? That's not a very good reason.
If an RPG developer like Obsidian or BioWare, in a future game, included extremely realistic physics interactions, Z-Axis exploration and all other sorts of complex interactivity like we see in other genres, one of two things would probably happen. One, the game would take six or seven years to come out (this is the unlikely outcome). After all, besides all of the features mentioned above, fans would still expect all of the RPG development systems, detailed characters and boatloads of branching dialogue, complete with choice and consequences, right? Second, the game comes out with all of those realistic action game interactions, but it is critically and commercially panned because including all of that stuff made the aforementioned RPG elements, that the company is known for in the first place, suffer. "Yeah, nice physics and stuff, but the story and characters blow, etc."
It all comes down to time and money, and the prioritization of that time and money.