I've made a bad habit of wandering into places that I'm pretty sure I wasn't meant to be yet. The first thing I did when I reached the first town was clear the temple. When I got the stronghold I went straight down the dungeon to level 8 or so (when I stopped being able to disarm traps or pick locks due to skill being too low). I still haven't been to Defiance Bay yet and I'm level 6. Stuff on the surface is just so trivial compared to those bastards under my castle.