It's still simulated violence, so if that's your objection to boxing, mma, or other martial arts you should take issue with fencing too. (Also as a former fencer I really wouldn't call it "tons of padding", it's just a jacket that can be surprisingly thin, an extra thin piece of nylon over your sword shoulder and pec, and sometimes a piece of plastic over your rib cage. Getting stabbed hard still leaves huge bruises)

With proper protection and awareness I don't think there's anything morally wrong with simulated fighting. All kinds of animals participate in play-fighting too, I don't think two kittens pouncing on each other is morally repugnant either. There's a clear difference between fighting for fun and fighting to maim/kill.

Can't remember where, but I was in a really good discussion with some other autistic people about playfighting as a kind of pressure release valve or other sort of stim and a solid handful of us realized that we'd had similar arrangements with friends when we're growing up — for me, a friend and I would take turns using an elbow to smack the other person's elbow until one of us gave up or we got bored. If anything it made me less interested in beating the shit out of someone.

As long as all parties are aware of any potential risks, safety measures are being taken, and participants have the ability to withdraw at any point, theres no moral issue inherent to the sport IMO.