how do you measure mtb frame size?

Centre of bottom bracket to top of seat tube (not top of clamp although that may be close, some clamps can stick up 1/2").

DO NOT use an existing frame size to buy a new bike, manufacturers other (much more important) dimensions are more critical to getting the frame size right. Even different bikes from the same manufacturer can vary a lot in sizing, between manufacturers is a pointless comparison.

You need to compare
Effective top tube length - from the centre of the top of the head tube horizontally back to the centre of the seatpost/seat tube. In addition add or subtract if one bike has an offset seatpost and the other inline as appropriate.
The next two are harder to measure
Reach - Horizontal distance from the centre of the bottom bracket to the centre of the bar mounting in the stem
Stack - Vertical distance between the same points
In addition you then need to allow for whether one has wider bars which needs a shorter stem.

Its reach and stack that define how well an MTB rides off roads when you're out the saddle (as its not a contact point) plus you then need to make sure the saddle is in the right place for pedally bits (which comes from ETT) - all assuming your current bike is actually the right size of course!

The height of the seat tubet is meaningless as your seatpost will give in excess of 150mm of adjustment, what is harder to adjust is the pedal to bars relationship by more than a few MM (20mm max on stack, very little on reach without impacting the steering) which is why they are more important.

Of course a bike with longer travel at the front/bigger wheels will tend towards a higher stack as the base of the headtube is higher and also the geometry (longer travel) will be more gravity orientated, if that's what you want you may then need a small reduction in reach to put your shoulders forward again.