I would say that freedom means doing what you want without harming others, and what they do is bothering you.
But you are right, it might be that some of them envy you your freedom - sometimes misery likes company, and 'if I cannot have it why should You?'
Rather than taking the example and getting themselves some freedom - which some people might actually do without your noticing it, ever think of that? ;-)