I hit this and worked around it by editing the conversation for clyver.
in Pillars of Eternity\PillarsOfEternity_Data\data\conversations\02_defiance_bay_first_fires\02_cv_clyver_rimgund.conversation
around line 2338 is the block for text node 54. What we'll do is let you access this even after he's said no before. This block has no check for relationship with the dozens, presumably because if the dozens like you he says no immediately, while if you're in hte middle of his quests he will say to ask later which is what this block is for.
Anyway, a bit further down, starting line 2406, there are ExpressionComponent blocks. You need to remove the entire block to do with the refused variable. I can't copy paste it because I deleted from mine and like a daring rebel (with a very fast internet connection to redownload the game) I didn't keep a back up copy.
The same principle should apply to the other groups if you prefer them but I don't know the nodes you would need to modify.