As the title implies, when you have anyone at the tower and, as part of the location starting condition, a Bandit is spawned and this bandit is evaded (whether by stealth or by using invisibility or a potion), the evasion does not count toward the daily quest. Evading any other sort of bane does seem to fulfil the quest condition, including banes spawned by ambushes, goblin raids and zombie hordes.