I personally think you've done exactly the right thing.

If your 'internal alarms' are sounding, you're absolutely right to abide by them. If the chap continues to follow you and annoy you, I would suggest getting the Police involved. On the basis of probabilities he is probably 'sad' rather than bad but it's time you got some external support. It's surprising what an informal visit from the Police can actually achieve.

I know it sounds silly, but if he continues to hassle you, please also make a note of when it happened as often the Police can then use that to challenge his behaviour. Most policemen and women will take your fears and concerns seriously.