It is irritating, and if you feel like giving a remark back, do, it is good against stomach ulcer ;-)
But I'd say let it go with that, you do not owe anyone any explanations, nor do you need to defend your lifestyle to others. It is none of their business.
I guess maybe the trick is to kind of not give such stupid people power over you by letting them upset you. Easier said than done, but fine when it works :-)