I'm having the same problem. We didn't come into contact with any of the priests, and we put on the robes. We also unequipped all of our weapons. The first acolyte we saw still turned hostile. So I restored from save and unequipped EVERYTHING -- all of our boots and stuff in addtion to the armor, all of our quick-slot items, the mage's grimoire, everything. We were wearing nothing but robes, and the first guy we saw still attacked us.
We have Durance with us, so I tried leaving him back in the room where we found the robes, in case he was known to be a priest of a rival god or something. But that didn't help, either.
I ended up slaughtering a bunch of priests, but I didn't WANT to. My character is a paladin of the Kind Wayfarers, and she likes being Benevolent, damn it!