I didn't say it wasn't self serving.
I was a favorite aunt. (Not a lot of competition, all 3 of my sister suck in various ways.) But that meant I skipped parenthood, and got the bennies of grandparenting --- sugar them out and send them home when they get whiny.
Generally, I think "you're selfish for not wanting kids" is a bit "women are only here to have kids" mixed with womb envy.