What you're not understanding is attraction is completely subjective looking from the outside looking in. How can I tell who or what the next man is attracted to, if not through his actions?
And yes you can control who you're sexually attracted to. You see a beautiful women with curves in all the right places, you ask around, and find out she's burning niggas. Are you sexually attracted to her? Hell no. Maybe your brother's/best friend's girl is someone you consider to be beautiful, or maybe your cousin, sister, daughter, niece, mother in law etc. are beautiful women. Are you sexually attracted to them? No, by choice.
Like I said earlier, we're not animals. Our mind supersedes all of those instinctive urges.