You make a good case but Im not sure I agree with the bolded part. Lest say you met identical twins, one male, one female. They look exactly alike except for maybe hair length and boobs (same face, height, weight, body type, etc...). You feel the only thing attracting you to one sex or the other is your upbringing?
If were to meet such a pair, I would only feel attracted to the female but I wouldn't really know why. But I'm sure various events in my very early life "conspired" to make me prefer the females of the species, rather than the males.
Also, note that when I say upbringing, I mean everything that makes a person what (s)he is today, except for genes.