Are you Catastrophizing? I didn’t know that word until recently, but I have met clients with this behavior. People who catastrophize imagine a lot of problems, even the most unlikely. For some, this condition places significant limitations on their lives.
The first step in fighting this is to become aware of the mechanism. When you are in a safe place and have time to think, consider which things you have recently decided not to do. Reflect on the reason you said no. If you find that you had a good reason, fine. You are rationally careful. If you no longer think saying no was necessary, then imagine a similar situation in the future. Imagine saying yes. That prepares you and lessens the risk of catastrophizing.