I am giving my baby his pacifier too much?

I feel like I am giving my baby the his pacifier too much. Throughout the day and night he will just be sucking on it. He does spit it out when he does want it anymore or falls asleep but most of the time I feel like he is attached to it. I regret introducing it to him now. When should I only give him his pacifier?