I thought it was AoE as well but even with new hirelings it won't change. I've had them start at green and turn red during a defend. Another one they were red from the start and another time I just visited the keep and one of the hirelings was red to me.
It's by no means game breaking just annoying.