You don't actually join Freedom or Duty, you just get friendly with them. And Skull wasn't the way to do that, anyway. He was connected to that, but kind of a sidequest on that path that you can pick up and do early. The actual quest for doing jobs for Duty comes from another guy who you get access to later, you'll get notified when.
However, by killing both Freedom and Duty guys, you just made things a bit harder for yourself in that regard.