In case this helps, I found that both clearing the first levels of the endless paths and upgrading your keep defences during act 1 decreases dramatically the frequency and the difficulty of random attacks later on. When I did this in every run the attacks were occasional and consisting of weak groups (oozes, looters etc). However, in a couple of runs I waited till act 2 to do those things and the difficulty and frequency of attacks changed dramatically (leaden key groups for starters, then lurkers and shades) - and that was regardless the level of the group, which was around 6 in all runs
