In general---as the aforementioned slogans indicate---ethical vegans do a mediocre job at best integrating their concerns about animals rights into these (equally?) critical moral issues (to be fair, those focused on other concerns aren't so cooperative either when it comes to animal rights). One reason for this reticence may be that incorporating other ethical concerns into our choice-making matrix blurs the ethical clarity that so many vegans take for granted. As much as we might like to think that eating ethically is simply about not eating animals, that's only the start of things. In fact, by making the noble decision to bother about animals at all, you open up many other cans of worms---and things can get sort of messy real quick. From this perspective, you can see why so many intelligent people go, "no, I don't want to know!"
Consider this scenario: you have a choice between eating roadkill and eating a plate of vegetables harvested by child slaves. If the slogan "I don't glare what you eat so long as it's not an animal product" holds, then you are forced by an overly rigid conceptualization of veganism to exploit child slaves rather than eat an animal that in no way was intentionally harmed for your consumption. You are, in other words, forced by your belief system to make an arguably immoral choice. That's an extreme case, but one could easily see how, as you leave the margins, the decisions become veritable toss-ups. For example, what if the choice was between eating oysters (questionably sentient critters) or a bowl of rice grown with water diverted from a subsistence village suffering a drought? Anyway, you get the idea.