Did a test with a dual wielding fighter with armored grace and a hired adventurer who was the same in every single aspect except not having armored grace.
The armored grace fighter wearing robe armor (-15% recovery speed) attacked at the same rate as the hired adventurer in no armor. When the hired adventurer wore robes, he attacked slightly slower than his armored grace counterpart in robes.
From what I've seen, it's a flat 16% reduction in recovery time with armored grace. Wearing robes would mean you are attacking at the same speed as being naked (15% recovery -16%=0). Padded armor would be a meager 4% recovery speed penalty (20%-16%=4%) and et cetera.
This makes armored grace fairly useful for those who want to absolutely maximize attack speed while still wearing protection, whether light, medium, or heavy armor.