The game is pretty lenient on structures, but you still need a supporting piece. Like the others said, if you remove something and everything falls apart, then that was a supporting piece. Try placing multiple scaffolding pieces around. Sometimes you can even swap support if you place a scaffold. Now if your doing something like building over the water or something, you may want to just pre plan your building.