Ghost Ship Sword Oath?

Is there any way to break your oath to return the blade? Any negative consequences? I tried just giving the sword to the ghost ship lady (nothing to say to the death guard in the crypt), exchanging the sword for the ship (the death guard in the crypt gets pissed and attacks), and going back with the sword after getting the ghost ship and I can choose to return the blade (or decline for now, which he suggests bad stuff will happen if I don't give it back soon). But I then went to Ukaizo with no apparent ill effects.

 

So rather curious about that.

I kept it and nothing bad happened.

Yeah, I thought I heard that something bad happened if you did and having played to try and break as many promises as possible with my super cruel run I was curious if anything was there. 

if you go back to ysemer and tell him you want to keep the blade a while longer, he's ok with that, oddly.

you ARE the herald of Berath, after all.