Vacationing with my family over the holidays led to very heated and divisive conversations about the sexual assault charges brought up against famed television icon Bill Cosby.

Some older members were conflicted on the nature of the cases. Many justifications were made — from personally discrediting some previous celebrity accusers to referencing the popular Quaalude drug culture of the '70s — but Cosby’s race was a more contentious topic.

“It seems like the media wants to tear down any positive image of a successful black man — even if Cosby did do something wrong — I feel sorry for his legacy,” said one family member. “When white men like Charlie Sheen or Woody Allen act up — they get a pass.”

Related: Bill Cosby defenders say race is huge factor in sexual assault charges

Excuse me, but Charlie Sheen having HIV doesn’t warrant him to get criticized harshly in the media — because that’s not criminal behavior. Even though there’s gossip that he may have hid his status to previous sexual partners — which is a crime in some states — such rumors haven’t gone any further than that.

As for Woody Allen’s longtime speculation of having sexual relations with his then underage adopted daughter — it’s no different than famed R&B star R. Kelly marrying the late singer Aaliyah at age 14. Even though Allen is white and Kelly is black, both were never legally convicted for their actions and went on to garner fame and fortune.

Related: The Ernest Opinion: Can there be a liberal arguments for guns?

Often with many celebrity sexual assault cases, financial settlements are made (such those made by R. Kelly for later child pornography charges) which implies that the color of money talks more than the color of skin in some cases. Which is why it's important to note that when it comes to matters of sexual assault, society must be mindful and sensitive when invoking race.

For one, both women of color and white women have alleged that Cosby has sexually assaulted them. Suggesting that his race is solely causing the backlash against him is dismissing the possible trauma of the very black women who might have been violated by this black man.

Yes, the criminal justice system has had a tumultuous reputation with handing various cases pertaining to black men. But the same level of distrust with the system can be said for their dealings with female rape victims as well…especially when it comes to allegations made against powerful, wealthy men.

Bill Cosby is no complete saint in any of this. In previous depositions made over a decade ago (that just became public last July) he revealed that he has paid off women he’s had sexual affairs with and previously used Quaaludes to help coerce sexual intercourse.

This isn’t to automatically imply that Cosby is guilty of the charges, but rather to fairly assess that his race isn’t the only reason why some are chastising America’s formerly favorite TV dad.