I don't think it's too much. Depends, I guess, on how willing you are to take on larger repairs as time goes on. Xtoolbox has more than that on his, I'm rolling past 120k with no worries, Rachel is over 160k and just drove to Moab to wheel, etc.
It's a good strong platform. Maintenance is key as is giving things the once-over more often.
Now if you found an low mileage 04 that you liked better than your X, that's one thing. I would think it perfectly reasonable, now knowing that the X will be your play rig of choice, to want to start with the newest, lowest mileage example. Especially if you have things you would change about your X now that you've had it a while (e.g., color, better seats, 5 sp. vs. auto, etc.). For example, if I had to do it all over again I, most likely, wouldn't have gotten the SC model and wouldv'e opted for a 5 speed.
If you're the type that likes to stay in tune with your rig and are happy with it, I say go for it.