Yeah, after a while I just went for PoTD upscaled. There's some fights that have been tougher - there was a giant black Ooze on some island, that I simply couldn't do at my level (I havent gone back to try now). The undead dragon was pretty hard to beat as well until I figured out I had to interrupt it to prevent it from casting its annoying defense spells. The island where I got my Druid the Spider Silk robe surprised me as well in terms of difficulty (also I think I might have gone there too early).
All in all though, I would say the game is not that difficult but I have made it more of a challenge by going to high level areas before I'm supposed to. Nonetheless, I enjoy it a lot and I'll definitely be doing a second playthrough with an evil/cruel character. So far I've really liked Rogue's class abilities so I'll probably try to do something with that. There's a lot of cool weapons and abilities across all the classes out there so ending up on one idea will very likely take a lot of time. (On a completely unrelated note - why have they given Ranger such bland abilities).
Back to difficulty - out of the recent games I've played in this genre, I would say this is easier that POE1 and Kingmaker were. Kingmaker's difficulty was not, however, handled in an enjoyable way. Deadfire has been thoroughly enjoyable so far (minus the frustration from Gorecci street :D).