no right or wrong answer
Correct, other than safety once back on the road.
Most of the time, for stock Xterra tires (265's/31") on stock wheels, airing down to 18/20 lbs.
off road is both safe and (as knightrider said) makes for a more comfortable ride. Carrying a mid quality ($100+ with batter alligator clip connectors, not 12V plug in connector) portable air compressor is also a great way to prepare for the end of the trail, so you may "air up" once you have completed your day.
"Airing down" does provide a larger traction patch for the tires, hence better "flotation" across sand and softer soils.
If you get a chance come out to a trail run and see the different systems that everyone employs and ask questions about how to "air down/up" for your best results. Also, watching a few videos on YouTube and surfing this Forum as well as other Xterra/off road Forums can help tremendously.
Being prepared for any off road situation is a good thing.
Then you will start the wallet evacuation on cool things like Staun deflators...and on board air...and CO2 fill tanks...etc.