I had a similar issue with limited options reported about stuck in ship combat (Stuck in Ship Combat Loop with 0 sail health). Hold then proceed half sail hold full sail hold hold half sail. I had to keep doing this over and over until I closed in on the other ship. The pain with this is having to hit continue multiple time in between cycles as if it just drags on.
 
So, after charging to the other ship and closing the distance the party vs party battle started on the ships
 
​The real issue
I defeated the opposing enemy. But that was it. I was stuck in battle. My character got knocked out during battle, but my other party members lived on. There were no other enemies to choose to battle. My player did not come back and the battle never ended.

Had the same problem once during a boarding. Also in some other places that had nothing to do with ship to ship battle (fightng the Dank Spore for example). So I think it's has not really anything to do with ships & stuff but with how the game determines that an encounter should end.

I had the same issue. When the battle ended, one of my crewmembers was in the process of casting a spell that never finished. I had to kill my own crewmember to end the battle.