Not to your own personal choice, but there absolutely is downside to stigma and helmet mandates. Studies show they reduce cycling numbers which means people drive instead, which increases death and injury, because spoiler, cars are dangerous.

Good article with references:

https://slate.com/technology/2023/01/bike-helmets-cyclist-deaths-do-you-need-to-wear.html

And more on other issues too with helmet mandates, like how police enforce… https://www.theguardian.com/us-news/2021/apr/20/us-helmet-laws-police-citations-cyclists-of-colour-homeless