This whole security system really seems quite odd... when I started building up my stronghold, my first thought was: prestige? Screw that, I need some BIG walls!!! So I just built every single building which brings security first (till about 35 security or so...), and just later the prestige stuff. Nonetheless, in late game, my stronghold got attacked by raider groups or something twice, and those jerks destroyed quite a lot of my buildings although my security was, so to speek, max Oo ?!?!