Same thing happens to me. I built my base on top of the wooden fence near the oak tree (where orb weavers are very common). I had build a storage container next to a lower fence piece, ants swarmed the bottom of the fence. Then I moved my base up to the taller post of the fence, with scaffolding for extra height, and they still managed to grab food from it. They teleported to the chest and started taking stuff from it. It basically forced me to keep all my food in my inventory which makes it more and more difficult to go out on supply runs because I keep three sets of armor for different scenarios and 4 food stacks of acorn bits, plus tools. Leaving me with little space left. It's super infuriating. I do appreciate that the base I'm at spiders and stuff aren't damaging the property and stuff but the ants are way too much. I really hope that this is fixed in the future.