Yes, it's simply a power dynamic. A man who doesn't feel he can overpower a woman will not harass her either. I am 5'2" and even in high school I'd have guys pick me up and swing me around simply because they could, despite my protests.
I was recently at a Meetup group where we started talking about abusive men. All of the women there agreed that having a man present who the abuser was afraid of went a long way in dissuading the abuse.
In addition, child abusers stop abusing their children when they grow big enough to "fight back".
When you start looking at how cowardly the behavior is, you realize just how disgusting the men who act out in this way actually are.
Thanks for sharing your story!