[Burden of Memory Debuff]

I was having the same problem and posted on reddit about it. since then it has disappeared whilst exploring on my ship for a couple of hours, it might be a time based or rest based debuff without a counter