So... you are lucky you were in a postion to stop him, once or twice. Did you report him to the police or social services? Seems like just a band-aid if not.
So... he accused you of being abusive, kind of the pot calling the kettle black. You know you do so for mutual pleasure, never when you're angry, AND can justify it because you have consent.
So what's the issue? Doesn't seem 'bothersome' to me to be accused by an abuser... more worrisome that you didn't help your former friend... or more importantly the woman he beat on.
Not pointing a finger, just food for thought should the opportunity rise again.