Don't start over (unless you really want too...) This exact situation is why you can hire a custom made NPC at ANY inn (and other places too). Also that temple basement is very tough unless you have the priest with you. Lots of high deflection, high DR shadows dealing freezing damage.
Even on "Normal" I struggled with only 3 member party, on Expert I struggled with 4 people in the party (none of them priests though...)