Reprocessing GPX files to exclude HR interference & correct distance.

wolfsbane2k

Hi.

I've got a problem where an intermittent Garmin HR monitor & speed sensor have completely messed up 60+ days of recording over the last year.
This is shown by :
1) reported HR hitting 254 BPM for 1 recording, and then returns to normal.
2) Reported distance significantly less than the correct distance due to intermittent speed sensor

I want to reprocess all of my data files to
1) Find and replace the HR measurement with the average of the measurement before and after the event.
2) Reprocess the reported distance to that recorded by GPS where the speed sensor has "disappeared" from recording.

Q- Has anyone else done this automatically, rather than mandocliatly going through each file? (commuting, so 2 files a day, 5 days a week)

I'm looking at GPX processing libraries that I can load into a basic C++ code, and then reprocess using some smart/fuzzy logic for the distances, then re-upload to garmin so that the garmin connect reports can be used properly.

Ta!

Jerry185

I understood the words 'Hi' and 'Ta'

chris_bass

If you export the files as gpx rather than tcx it only contains the gps points so will not have any of your sensor data included. If you reupload this it will then use the gps point to calculate and give the correct distance. Won't help with your dodgy heart rate readings though. I don't know how to do any of that by bulk though so would need to do each one manually.

Were you using anything bluetooth near the sensors? i find that used to make mine intermittently drop out.

wolfsbane2k

Na, it's not BT related - the HRM is 10 years old, has been swimming many times, and needed a good clean, or more likely replacement. The speed sensor problem was caused by a slightly slipped magnet or sensor so that it would miss when traveling at speed.

The gpx files I've copied straight from the device ( 705) have all the sensor data in as well, so was going to play with those then completely re-upload if possible.
I don't "just" want to use the GPS data, as in some instances, such as going into a shop with the tracker still active, goes haywire and adds a mile over a 20 minute shop, which then goes the other way.

I've done a bunch by hand so know the tell tale signs of the problem so its just a matter of coding up what steps I'm taking, but it's getting it into a easy to access array first, then I can try different smoothing functions.

Matthewfalle

You do know that this wouldn't happen if you used vinyl?

chris_bass

Sounds like you messed up the gps data by not stopping it when you were stopped and messed up the sensor data due to dodgy sensors/user error.

might be worth just setting things up right now and leaving the old stuff as is.

wolfsbane2k

Chris Bass wrote:

Sounds like you messed up the gps data by not stopping it when you were stopped and messed up the sensor data due to dodgy sensors/user error.

might be worth just setting things up right now and leaving the old stuff as is.

That's exactly it.
I'm kinda considering this an recovery project while on enforced leave due to operation. It's either that or fill out my chartership application...

What's annoying me the most about the hr peaks is the screwup it's having on my fitness metrics - I want to plot HR improvements over a specific segment over a 6 month period and it's screwing it all up.
And I've lost about 400 miles of riding for b to the dodgy speed sensor, so want to recover that as it's 3 weeks with of riding.

wongataa

Using something like 'Fit File Repair Tool' is probably the best solution. With that you can use the GPS points to recalculate the distances and edit out errors due to stops and edit HR data. It is not free though.

rolf_f

Wolfsbane2k wrote:

Chris Bass wrote:

Sounds like you messed up the gps data by not stopping it when you were stopped and messed up the sensor data due to dodgy sensors/user error.

might be worth just setting things up right now and leaving the old stuff as is.

That's exactly it.
I'm kinda considering this an recovery project while on enforced leave due to operation. It's either that or fill out my chartership application...

What's annoying me the most about the hr peaks is the screwup it's having on my fitness metrics - I want to plot HR improvements over a specific segment over a 6 month period and it's screwing it all up.
And I've lost about 400 miles of riding for b to the dodgy speed sensor, so want to recover that as it's 3 weeks with of riding.

If you want a project then you could process the data into Excel. Very easy then (using sort function) to eliminate outliers and you can do much more than the online tools in terms of showing the data in different ways.

thomasmorris

Golden Cheetah has a data editing tool. Still very time consuming as you have to edit file by file. But it will search for errors for you, and then you can just copy and paste over the top. I think you can do a 'find and replace' type action for anything over 200 bpm etc.


I understand the frustration with the HR as I've had a similar thing before, but the real trick is just editing the one or two files then fixing the cause of the problem. I've also had the the same thing with dodgey power meter which screws the power curve and best efforts.

You'll never get that data back, but you'll be able to remove the big spikes that skew training stress (rides where you've done 20 minutes over 200 bpm due to static).

The distance I'd just leave as it is, or use GPS. Distance as a metric is fairly useless anyway, as not all miles are equal in effort. If I had a GPS jump which added on a few hundred or thousand KM i'd edit it, otherwise a KM here or there isn't worth worrying about.

ben-----

> but it's getting it into a easy to access array first, then I can try different smoothing functions.

I could easily do that for you. If you post one block of the gpx code you've got, that is one location point (which'll amount to about, very roughly, 10 lines of xml data), then a version of how you would want that…