At the moment the current system is really brutal in that a character not built around Armour penetration (AP) could be stuck doing 30% damage for entire battle.
However this is only because currently Armour and AP are fixed values once buffs and afflictions are in effect.

If AP became a range then the effects of armour would still feel very different than accuracy and give a sense of realism. (A stiletto could have AP 1-12, mace 6-9) 
This should also make it easier to balance AP talents. (If +1 AP proves to strong to pass up then the devs can make the variable wider and increase all armour values)

This could also mean that every increase or decrease in AP will always effect a battle without becoming the main concern.

 

I imagine some effects should be calculated once to prevent lag from calculations (Damage over time, AOE etc.)

Keep up mate, you're a bit behind.

To be honest, I think your (OP) suggestion would only serve to i) complicate combat reporting even more, ii) increase RNG, and iii) make the average swing more ineffective. Specifically:

 

i) The AP/Armour system was introduced in order to make combat easier to understand and more tactically rewarding. The idea was that, once you see that a character isn't effectively attacking (which is very easy to see in the new system) you practically have to switch out tactics. In your proposed system it will be useless to notify a player that an attack is ineffective, because maybe it was just a bad roll. And maybe the next attack is just a bad roll... so at what time (unless you are extremely involved in the combat and keep an eagle eye on the log) do you switch out tactics? In addition, I would guess that the average player would then ignore this mechanic for most combats and just keep on swinging without changing tactics, similar to the way DR was generally ignored (unless ridiculously high/immune) by most players in PoE 1.

 

ii) Your system includes an entirely extra roll for each attack, which means RNG will play an even bigger roll in the way battles proceed, which generally reduces the need for tactics in a game. It also feels bad. Nothing is worse than the feeling when you buff someone two or three times, but then they miss their AP roll 3 times in a row. While it already happens with accuracy rolls + damage rolls, including a third option for these things to happen is only going to retract from the game.

 

iii) Finally, related to the point above, including an additional roll will make, on average, all swings less effective, since the probabilities are entirely uncorrelated. For instance, the probability of getting a damaging, AP>Armour critical strike will be much less likely, as you have to have 3 high-rolls in a row. In order to balance for this, all ranges (such as crit range) would have to be increased, which means you would crit much more often but it won't result in any significant damage. As an example, in your proposed system:

 

You swing the first time. You hit critically (Acc >> Defl). You roll great on your damage. You roll poorly on your AP (despite +crit bonus), = min dmg

You swing a second time. You hit normally (Acc >= Defl). You roll ok on your AP. You roll crap on your damage, = min dmg

You swing a third time. You roll great on your damage and AP, but it was a miss (although dmg and AP wouldn't have been rolled in this case)

 

You get the idea. Ideally, a designer wants as few rolls as possible without creating a too simple system. Removing one of the rolls, but keeping the effect (e.g. Gated penetration system) is a way to add complexity without the problem of increasing RNG and having to balance around it. In the current version of the beta, with spells accuracy which is reduced, offensive spells are entirely too hit-or-mis (Hah) to consider using for important fights. Imagine having even another RNG range to deal with and balance on top of this.

 

Sorry for pulling your idea apart - it is meant to be constructive feedback.

 

Now, with grazes back in, and the under-pen penalty continuous (although quantized) rather than binary, the new system might work well - we'll know when we get to play around with it.