I was running into this issue, so now I basically rub the screen where the dice are. On another semi related note, sometimes I will try to move a card from my hand to the banish area and I won't realize that I haven't grabbed the card until it is too late and end up rolling the dice.