Found the issue.
I ran a save just before opening the door to steal the book.
Stealthed (but with red eyes, meaning you are seen), you can enter without triggering the hostile flag. But if you leave and enter again, it will trigger the hostile flag. This is the true bug.
I tried it without stealth, and it triggered the hostile flag immediately.
I would suggest the fix to not allow you to enter the door in stealth, especially with red eyes as it shouldn't be allowed anyway.
Welp... time to replay two hours -.- Hope this helps dev. The issue definitely resides upon entering the door in stealth.