It is extremely easy to say, "No."
The only reason we usually THINK it is hard is because we have bought into the idea that we have to tell people why we cannot do something. As a born "people pleaser" I used to be plagued with this problem myself until I learned how to say "No" without giving a reason.
Generally, I have found that just saying "No" works quite well:
"Do you want to go to town this afternoon?"
"No, not today."
If someone press with a "Why not?" I may say,"I have other plans," or if pushed, "I'm sorry, that's a private matter I really can't discuss," but I avoid to tell exactly what plans or commitments I have.
Even if people press for a reason, once you realize that you do not owe anyone an explanation for your choices it helps.
Over time I have learned how to answer some people in ways that avoid their unreasonable demand for an excuse while seeming to do so to some extent.
"Thanks for asking. I wish I had more time for things like that."
"Thanks for asking, but my funds are all committed right now."