It takes that long if you don't know how to use a map, or have no sense of direction He was probably going around in circles. "
As a side note: With the right gear (that you can get/make) fairly early) you can cross the island in about 3 minutes (and I'm not talking about the suicide jump spell)