I was wondering if it was just something I was doing wrong, but I see the same problem.
I got this answered in another forum:
You can go back to Defiance Bay at some point.
Specifically, when you walk into Twin Elms for the first time, the gate guards mention that the riots in Defiance Bay have subsided and that the city gates are open again. It's their not-subtle way of saying "go away, estramor."