Am I so wrong to shield son from my brother-in-law?

My husband's family is a total nightmare and I spend my life wishing we could just cut them off and never see them again.

- 

The problem is my husband's younger brother, who’s really messed up and lives with his parents. He’s been in trouble with the police more than once and has had a drug and alcohol problem for years.

His parents are way too soft with him and make excuses for him all the time. We don't want to visit them with our son while my brother-in-law is living there, but they take great offence at this and have made it our issue - somehow we're the ones who are unreasonable!