UNICSV to GPX can't handle 5 Hz GPS data

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

UNICSV to GPX can't handle 5 Hz GPS data

Dan Yuknavage

Seeking help with a universal CSV to GPX conversion.

Issue is UNICSV is not handling the time stamp properly with 5 Hz GPS data. The lat, lon and date are correct in the GPX output but the time is incorrect. There is no problem with 1 Hz GPS data. The GPX output file has the correct time stamp.

 

Using GPSBable version 1.5.1 and copied the information below from the 'About GPSBabel' window

(Using backend 1.5.1)

Installation ID: {8754d53d-111b-4776-927c-c78dfa155cb7}

 

The Input data file starts with this:

lat,lon,date,time

33.78480148,-117.47435,2014/9/17,13:20:20.4

33.78480148,-117.4743424,2014/9/17,13:20:20.6

33.78480148,-117.4743424,2014/9/17,13:20:20.8

33.78480148,-117.4743424,2014/9/17,13:20:21.0

33.78480148,-117.4743424,2014/9/17,13:20:21.2

33.78480148,-117.4743347,2014/9/17,13:20:21.4

33.78480148,-117.4743347,2014/9/17,13:20:21.6

33.78480148,-117.4743347,2014/9/17,13:20:21.8

33.7848053,-117.4743271,2014/9/17,13:20:22.0

33.7848053,-117.4743271,2014/9/17,13:20:22.2

33.7848053,-117.4743271,2014/9/17,13:20:22.4

 

 

The GPX output is getting a scrambled time stamp:

<?xml version="1.0" encoding="UTF-8"?>

<gpx version="1.0" creator="GPSBabel - http://www.gpsbabel.org" xmlns="http://www.topografix.com/GPX/1/0">

  <time>2014-09-26T17:31:19.462Z</time>

  <bounds minlat="33.782070160" minlon="-117.476020800" maxlat="33.784805300" maxlon="-117.473014800"/>

  <trk>

    <trkseg>

      <trkpt lat="33.784801480" lon="-117.474350000">

        <time>2014-09-17T20:27:00Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474342400">

        <time>2014-09-17T20:30:20Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474342400">

        <time>2014-09-17T20:33:40Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474342400">

        <time>2014-09-17T20:20:21Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474342400">

        <time>2014-09-17T20:23:41Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474334700">

        <time>2014-09-17T20:27:01Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474334700">

        <time>2014-09-17T20:30:21Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474334700">

        <time>2014-09-17T20:33:41Z</time>

      </trkpt>

      <trkpt lat="33.784805300" lon="-117.474327100">

        <time>2014-09-17T20:20:22Z</time>

      </trkpt>

      <trkpt lat="33.784805300" lon="-117.474327100">

        <time>2014-09-17T20:23:42Z</time>

      </trkpt>

      <trkpt lat="33.784805300" lon="-117.474327100">

        <time>2014-09-17T20:27:02Z</time>

 

The attached files are the two Input CSV and the bad GPX output.

CoronaSXtrksDateTime.csv is the input file with 5 Hz GPS sampling rate.

CoronaSXtrksDateTime.gpx is the output file with the bad time stamp.

CoronaSXtrksDateTime1Hz.csv is the same input file with a 1 Hz time stamp that translates to GPX with no errors.

The problem is repeatable with any CSV file using the 5 Hz GPS data.

Please let me know if you need more information.

Thanks,

Dan Yuknavage

Yamaha USA Testing

714 761-6118

 


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc

CoronaSXtrksDateTime.gpx (44K) Download Attachment
CoronaSXtrksDateTime.csv (18K) Download Attachment
CoronaSXtrksDateTime1Hz.csv (17K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: UNICSV to GPX can't handle 5 Hz GPS data

Robert Lipe-4
Thanx for a good report.

You're right. When we beefed up our sub-second support for 1.5, unicsv didn't get the attention it deserved and it has time_t's scattered All Over.  It does lots of gnarly guessing of time and date formats, so it needs more than a casual sprinkling of QDateTime seasoning.

I'll put this on my list of things to fix unless someone else wants to grab this first.

RJL

On Tue, Sep 30, 2014 at 2:51 PM, Dan Yuknavage <[hidden email]> wrote:

Seeking help with a universal CSV to GPX conversion.

Issue is UNICSV is not handling the time stamp properly with 5 Hz GPS data. The lat, lon and date are correct in the GPX output but the time is incorrect. There is no problem with 1 Hz GPS data. The GPX output file has the correct time stamp.

 

Using GPSBable version 1.5.1 and copied the information below from the 'About GPSBabel' window

(Using backend 1.5.1)

Installation ID: {8754d53d-111b-4776-927c-c78dfa155cb7}

 

The Input data file starts with this:

lat,lon,date,time

33.78480148,-117.47435,2014/9/17,13:20:20.4

33.78480148,-117.4743424,2014/9/17,13:20:20.6

33.78480148,-117.4743424,2014/9/17,13:20:20.8

33.78480148,-117.4743424,2014/9/17,13:20:21.0

33.78480148,-117.4743424,2014/9/17,13:20:21.2

33.78480148,-117.4743347,2014/9/17,13:20:21.4

33.78480148,-117.4743347,2014/9/17,13:20:21.6

33.78480148,-117.4743347,2014/9/17,13:20:21.8

33.7848053,-117.4743271,2014/9/17,13:20:22.0

33.7848053,-117.4743271,2014/9/17,13:20:22.2

33.7848053,-117.4743271,2014/9/17,13:20:22.4

 

 

The GPX output is getting a scrambled time stamp:

<?xml version="1.0" encoding="UTF-8"?>

<gpx version="1.0" creator="GPSBabel - http://www.gpsbabel.org" xmlns="http://www.topografix.com/GPX/1/0">

  <time>2014-09-26T17:31:19.462Z</time>

  <bounds minlat="33.782070160" minlon="-117.476020800" maxlat="33.784805300" maxlon="-117.473014800"/>

  <trk>

    <trkseg>

      <trkpt lat="33.784801480" lon="-117.474350000">

        <time>2014-09-17T20:27:00Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474342400">

        <time>2014-09-17T20:30:20Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474342400">

        <time>2014-09-17T20:33:40Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474342400">

        <time>2014-09-17T20:20:21Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474342400">

        <time>2014-09-17T20:23:41Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474334700">

        <time>2014-09-17T20:27:01Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474334700">

        <time>2014-09-17T20:30:21Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474334700">

        <time>2014-09-17T20:33:41Z</time>

      </trkpt>

      <trkpt lat="33.784805300" lon="-117.474327100">

        <time>2014-09-17T20:20:22Z</time>

      </trkpt>

      <trkpt lat="33.784805300" lon="-117.474327100">

        <time>2014-09-17T20:23:42Z</time>

      </trkpt>

      <trkpt lat="33.784805300" lon="-117.474327100">

        <time>2014-09-17T20:27:02Z</time>

 

The attached files are the two Input CSV and the bad GPX output.

CoronaSXtrksDateTime.csv is the input file with 5 Hz GPS sampling rate.

CoronaSXtrksDateTime.gpx is the output file with the bad time stamp.

CoronaSXtrksDateTime1Hz.csv is the same input file with a 1 Hz time stamp that translates to GPX with no errors.

The problem is repeatable with any CSV file using the 5 Hz GPS data.

Please let me know if you need more information.

Thanks,

Dan Yuknavage

Yamaha USA Testing

<a href="tel:714%20761-6118" value="+17147616118" target="_blank">714 761-6118

 


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc



------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|

Re: UNICSV to GPX can't handle 5 Hz GPS data

Robert Lipe-4
I started the work on this this last weekend and just finished checking in the rest of the fixes to make your sample work.  I also distilled your example into a regression test so we don't break it in the future.

It'll be in the next (currently unscheduled) release and is available in SVN now if you want to build from source.

Thanx,
RJL

On Tue, Sep 30, 2014 at 2:59 PM, Robert Lipe <[hidden email]> wrote:
Thanx for a good report.

You're right. When we beefed up our sub-second support for 1.5, unicsv didn't get the attention it deserved and it has time_t's scattered All Over.  It does lots of gnarly guessing of time and date formats, so it needs more than a casual sprinkling of QDateTime seasoning.

I'll put this on my list of things to fix unless someone else wants to grab this first.

RJL

On Tue, Sep 30, 2014 at 2:51 PM, Dan Yuknavage <[hidden email]> wrote:

Seeking help with a universal CSV to GPX conversion.

Issue is UNICSV is not handling the time stamp properly with 5 Hz GPS data. The lat, lon and date are correct in the GPX output but the time is incorrect. There is no problem with 1 Hz GPS data. The GPX output file has the correct time stamp.

 

Using GPSBable version 1.5.1 and copied the information below from the 'About GPSBabel' window

(Using backend 1.5.1)

Installation ID: {8754d53d-111b-4776-927c-c78dfa155cb7}

 

The Input data file starts with this:

lat,lon,date,time

33.78480148,-117.47435,2014/9/17,13:20:20.4

33.78480148,-117.4743424,2014/9/17,13:20:20.6

33.78480148,-117.4743424,2014/9/17,13:20:20.8

33.78480148,-117.4743424,2014/9/17,13:20:21.0

33.78480148,-117.4743424,2014/9/17,13:20:21.2

33.78480148,-117.4743347,2014/9/17,13:20:21.4

33.78480148,-117.4743347,2014/9/17,13:20:21.6

33.78480148,-117.4743347,2014/9/17,13:20:21.8

33.7848053,-117.4743271,2014/9/17,13:20:22.0

33.7848053,-117.4743271,2014/9/17,13:20:22.2

33.7848053,-117.4743271,2014/9/17,13:20:22.4

 

 

The GPX output is getting a scrambled time stamp:

<?xml version="1.0" encoding="UTF-8"?>

<gpx version="1.0" creator="GPSBabel - http://www.gpsbabel.org" xmlns="http://www.topografix.com/GPX/1/0">

  <time>2014-09-26T17:31:19.462Z</time>

  <bounds minlat="33.782070160" minlon="-117.476020800" maxlat="33.784805300" maxlon="-117.473014800"/>

  <trk>

    <trkseg>

      <trkpt lat="33.784801480" lon="-117.474350000">

        <time>2014-09-17T20:27:00Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474342400">

        <time>2014-09-17T20:30:20Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474342400">

        <time>2014-09-17T20:33:40Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474342400">

        <time>2014-09-17T20:20:21Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474342400">

        <time>2014-09-17T20:23:41Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474334700">

        <time>2014-09-17T20:27:01Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474334700">

        <time>2014-09-17T20:30:21Z</time>

      </trkpt>

      <trkpt lat="33.784801480" lon="-117.474334700">

        <time>2014-09-17T20:33:41Z</time>

      </trkpt>

      <trkpt lat="33.784805300" lon="-117.474327100">

        <time>2014-09-17T20:20:22Z</time>

      </trkpt>

      <trkpt lat="33.784805300" lon="-117.474327100">

        <time>2014-09-17T20:23:42Z</time>

      </trkpt>

      <trkpt lat="33.784805300" lon="-117.474327100">

        <time>2014-09-17T20:27:02Z</time>

 

The attached files are the two Input CSV and the bad GPX output.

CoronaSXtrksDateTime.csv is the input file with 5 Hz GPS sampling rate.

CoronaSXtrksDateTime.gpx is the output file with the bad time stamp.

CoronaSXtrksDateTime1Hz.csv is the same input file with a 1 Hz time stamp that translates to GPX with no errors.

The problem is repeatable with any CSV file using the 5 Hz GPS data.

Please let me know if you need more information.

Thanks,

Dan Yuknavage

Yamaha USA Testing

<a href="tel:714%20761-6118" value="+17147616118" target="_blank">714 761-6118

 


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc




------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc