Pretty much just the title. Every Sunday we get together and play for 4 hours and honestly there are like 100 things I'd rather do. I love DnD shows and content (which I acknowledge aren't actually existing DnD) so I keep getting suckered into giving it a try.
I just hate to quit mid session because thats rude and is a pain to everyone else but also I don't want to spend 4 hours every Sunday on my day off doing something I'm not enjoying.
How do I end this lightly? Can I?
No, your anxiety is rational because it will effect others and their is a risk of a negative reaction. What is irrational is jumping to the conclusion that there will be a negative reaction, assuming that people aren't understanding and gracious, and probably labeling yourself with negative titles for the simple act of meeting your own needs and desires.
The most likely and common scenarios are not extreme outcomes, your fear is narrowing your focus on these things. It will not go horribly, and it might not go 100% smooth, but it will be something that you can handle.