Not really a man (Agender and Intersex) but I am Male so I feel like I can answer this one.
I have a developmental condition which affects my body's ability to produce androgens so for me I wouldn't be much different aside from genitals since I never finished puberty and likely wouldn't have then either.
Not as big of a setback or difference than it would be for someone else but not being able to pee while standing would be a very big negative for me and I'm not really sure what I would do about that (probably get a Phalloplasty so I can pee properly again).
If I woke up as female, I am Agender so I'm not a woman because I do not identify as one.
Essentially though yes, yes I would, because even though I am Agender having a penis is something that feels right for me.
I know that I'm going to get shit for this from people who don't understand why something like this is important to me, but I'm used to it, NonBinary identities are much less recognized and often stigmatized (even in safe spaces 😔).