He always rolls 1 or 2 if I defeat him.  The moment I get the 1,1,2,3 on 4 dice, then he rolls 6.  This was the 8th fight for that single encounter.  I cannot figure out a way to beat the RNG. 

Keep your fingers crossed and hope. Or chose a hero who can beat him without any help... eventually he will come down. But yeah, he can be real pain in the... I Have strugled with him a couple of times

But They say, the Taller, the harder it hits down!

You probably don't beat him, unless the character fighting him can reliably clear the check without assistance. But the odds of him rolling a 1 or 2 eight times in a row are somewhere south of 1 in 10,000, so if it happens take solace in knowing it was just horrid luck.