Actually yeah. Played a long game a couple of weeks ago vs an UD on Plunder Isle who went Wyrms. I went Mountain Giants and Dryads and it actually worked pretty well. I ultimately lost though, because I never got enough money to rebuild my expo in the upper left corner after my other two mines were dried out.
But it should be fully possible to defeat Wyrms as a NE player. As you probably know dryads are immune to magic damage so they won't take damage from wyrms, and with the Warden as a semi-tanker to finish off the fiends with Fan of Knives and Shadowstrike it shouldn't be a problem if you micro well. Producing 1 bear to get 25% more attack also helps a lot, as well as having healing and armor scrolls on your hero. And the Panda is a must as second hero on tavern maps.