As someone who is well out of the range where people need to ask for ID when I want something, I don't concern myself with my age and frequently forget since nobody asks.
On official forms, they want to know my date of birth, not my age. Everyone I know socially, doesn't care or already knows an approximation of it, thus I'm never asked. All my work colleagues are interested in is getting the job done, not how many years I've been in the workforce/how many years till I get to retirement age/whatever.
I don't think about my age for more than a few hours a year at most. I don't even celebrate my birthday. So the years come and go, and I think about how old I am so little that I often don't even know how old I am.
You're not wrong that it's not hard to remember, but when you think about it as little as I do, you tend to lose track.... So when someone asks me how old I am, I often don't reply because I'm too busy doing math in my head to try to figure it out, quietly hoping they'll move on from the question and I won't have to answer it.