Hmmm. I know quite a few very fit women, but it is also no secret that women in general can't compete on an athletic basis with men. Testosterone does a lot to increase muscle mass, and decreasing testosterone will certainly result in a significant loss of that muscle mass. If you have less muscle what you do have has to work harder and will tire more quickly.
I used to push my body very hard, especially when I was in the service. I've stopped most of that (because of injury) but I'm also sure that actively feminizing myself will accelerate the muscle loss.
I don't know about the brain re-wiring causing a lack of stamina. I haven't encountered that yet but then I still can't push myself very hard. I have always heard, however, that many women could actually tolerate pain and discomfort better than men, to prepare then better for child-birth. Still, time will tell!
Don't discount the getting older part also. It sucks!