Actually I no longer think this is a bug by itself, I suspect I entered the manor earlier and triggered the hostile reaction. I've tried different older saves and there is indeed a moment after which the saves have a new journal entry about finding the manor entrance and imps are hostile after that.