Trust your own instincts.

If they are telling you this relationship is wrong, not listening to them would be the worst thing you could do.