Gilded Emnity and the Damage Bonus I’d Sworn Enemy

[Torm51]

Do you just get the armor buff and lose out on the damage bonus? When I cast sworn enemy I see my armor buff but don’t see a de buff on the sworn target...maybe I saw that wrong??