Throw a few goblins their way. Maybe a Rhino.

Or step it up a notch. (And ya gotta wait until this comes up in conversation.)

Session 1: My Chaos Paladin tells party his goddess' name is Beth.

Session 2: Find magic sword, name it Beth.

Session 3: Tell party my goddess wants to be called Kate now. (Chaotics, ya know?)

Session 4: Explain that the goddess is OK if I still call my sword Beth, but I should name the next one Kate.

Session 5: Utter a vow of enmity: "For Kate's sake I stick my last Beth at thee!"

Send a couple of yochlols their way. That ought to get rid of his smitey sense. Lolth doesn't take kindly to usurpers.