Man, I wish I could blame all my personal problems on other people. Doesn't that sound awesome? It's so liberating. People avoid you not because you're a large bulky pink-haired homeless man in assless chaps, but because they somehow instinctively sniff out the Other in you.
What's wrong with the confederate flag?
And, of course, Hooves is assuming she's a white transgendered woman. There's no way he actually knows any of these characterists. After all, no one looking at Phil would see anything other than a smelly white homeless man. (Which is probably why she moved in the first place)
Every other phobia than xenophobia is a more likely reason for not wanting to sit next to him.
Now, now... I doubt very much gynophobia would play a part, either.