bhutchison is absolutely right. Just tried this out myself and it works. As soon as I disabled my primary network adapter (my secondary has no affect) the game would run smoothly, without any delays. Thanks for this suggestion, bhutchison. Hopefully this can help narrow the bug down.
Interestingly, I feel, I experienced a similar bug with Kerbal Space Program, in that its initial launch would take upwards of 30 minutes if the primary network adapter was not disabled before launching the game. Since both Kerbal and Pillars of Eternity share the Unity engine, could this perhaps be a Unity engine bug? I haven't played Kerbal in some time, and the last I heard was that the developers there were trying to work with the Unity devs to sort out the problem - perhaps there is a crossover bug here thanks to the engine?
Funny you should say that, I picked up KSP a month ago and was astonished by the launch time. It took ages to start the game up, so it must be a unity related problem indeed.