Quantcast

CSV output formatting

classic Classic list List threaded Threaded
7 messages Options
SRE
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

CSV output formatting

SRE
This page has the "xcsv" format, which doesn't seem to be an option in the GUI:
        http://www.gpsbabel.org/htmldoc-1.5.0/gpsbabel-1.5.0.pdf

The documentation for "universal csv" says it can read and write waypoints, but
when I feed GPSBabel a GPX file and ask for Universal CSV output I get nothing
at all. There IS output for regular CSV, but it doesn't have all the fields I need.

So is there an easy way to choose which fields are output in a CSV file?
(like short name, comment, lat, lon) It seems like I'm missing somthing obvious!

Steve


------------------------------------------------------------------------------
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CSV output formatting

Robert Lipe-4
xcsv is not usable from the gui.  that's the "behind the scenes" format that's used for the Style system.  http://www.gpsbabel.org/htmldoc-development/styles_intro.html

Unicsv outputs everything it knows about; you can't control the order.

If you want total control of the xcsv, write your own style description as described above and use the xcsv option from the command line.

On Sat, Nov 7, 2015 at 1:50 PM, SRE <[hidden email]> wrote:
This page has the "xcsv" format, which doesn't seem to be an option in the GUI:
        http://www.gpsbabel.org/htmldoc-1.5.0/gpsbabel-1.5.0.pdf

The documentation for "universal csv" says it can read and write waypoints, but
when I feed GPSBabel a GPX file and ask for Universal CSV output I get nothing
at all. There IS output for regular CSV, but it doesn't have all the fields I need.

So is there an easy way to choose which fields are output in a CSV file?
(like short name, comment, lat, lon) It seems like I'm missing somthing obvious!

Steve


------------------------------------------------------------------------------
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc


------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
SRE
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CSV output formatting

SRE
At 03:18 PM 11/7/2015, Robert Lipe wrote:
>Unicsv outputs everything it knows about; you can't control the order.

I tried this:
        > gpsbabel -D9 -s -w -r -t -i gpx -f TestCSV.gpx -o unicsv -F TestCSV.csv      
        options: module/option=value: gpx/snlen="32" (=default)
        options: module/option=value: unicsv/datum="WGS 84" (=default)
        options: module/option=value: unicsv/prec="6" (=default)
        cet_util: Converting from "UTF-8" to "US-ASCII", done.
        GPSBabel Version: 1.5.2
        Translation successful

But I got an empty file! With only three header columns (number, lat, lon)
even though the input GPX has elevation, time, name, comments, etc.

Am I doing something wrong? I expected a CSV file with every known field
as a separate column. Sorry for being a dunce if this is obvious...

Steve

------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc

TestCSV.gpx (6K) Download Attachment
TestCSV.csv (23 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CSV output formatting

Robert Lipe-4
I'd have hoped you'd have gotten an error from that malformed TestCSV.gpx input file. It had a bunch of random characters and line breaks in places that I don't think are legal

 emas/T
ripMetaDataExtensions/v1 http://www.garmin.com/xmlschemas/TripMetaDataExtensionsv1.xsd http://www.garmin.com/xmlschemas/ViaPointTransportationModeExtensions/v1 http://www.garmin.com/xmlschemas/ViaPointTransportationModeExtensionsv1.xsd http://www.garmin.com/xmlschemas/CreationTimeExtension/v1 http://www.garmin.com/xmlschemas/CreationTimeExtensionsv1.xsd http://www.garmin.com/xmlschemas/AccelerationExtension/v1 http://www.garmin.com/xmlschemas/AccelerationExtensionv1.xsd http://www.garmin.com/xmlschemas/PowerExtension/v1 http://www.garmin.com/xmlschemas/PowerExtensionv1.xsd http://www.garmin.com/xmlschemas/VideoExtension/v1 http://www.garmin.com/xmlschemas/VideoExtensionv1.xsd" xmlns="http://www.topografix.com/GPX/1/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:wptx1="http://www.garmin.com/xmlschemas/WaypointExtension/v1" xmlns:gpxtrx="http://www.garmin.com/xmlschemas/GpxExtensions/v3" xmlns:gpxtpx="http://www.garmin.com/xmlschemas/TrackPointExtension/v1" xml!
 ns:gpx
x="http://www.garmin.com/xmlschemas/GpxExtensions/v3" xmlns:trp="http://www.garmin.com/xmlschemas/TripExtensions/v1" xmlns:adv="http://www.garmin.com/xmlschemas/AdventuresExtensions/v1" xmlns:prs="http://www.garmin.com/xmlschemas/PressureEx


If I de-garminify the header so the file starts off

<?xml version="1.0" encoding="utf-8"?><gpx creator="Garmin Desktop App" version="1.1"  xmlns:gpxx="http://www.garmin.com/xmlschemas/GpxExtensions/v3" xmlns:trp="http://www.garmin.com/xmlschemas/TripExtensions/v1" xmlns:wptx1="junk" xmlns:ctx="junk" >
  <metadata>

$ ./gpsbabel -i gpx -f ~/Downloads/TestCSV-fixed.gpx  -o unicsv -F -
No,Latitude,Longitude,Name,Altitude,Description,Symbol,Date,Time
1,49.635660,-114.587120,"TC ABC MS",891.5,"26-AUG-13 13:22","Flag, Green",2014/06/29,01:22:29
2,56.370833,-119.924133,"TC ALCES",,"TCP ALCES (?)","Flag, Green",,
3,56.302533,-112.310000,"TC ALGARLK",,"TCP ALGAR LAKE 123.2","Flag, Green",2011/11/02,16:16:55

Seems legit.


It looks like we may have improved the GPX error handling since 1.5.2.  Your input file is definitely toast.

RJL


On Tue, Nov 17, 2015 at 7:38 PM, SRE <[hidden email]> wrote:
At 03:18 PM 11/7/2015, Robert Lipe wrote:
>Unicsv outputs everything it knows about; you can't control the order.

I tried this:
        > gpsbabel -D9 -s -w -r -t -i gpx -f TestCSV.gpx -o unicsv -F TestCSV.csv
        options: module/option=value: gpx/snlen="32" (=default)
        options: module/option=value: unicsv/datum="WGS 84" (=default)
        options: module/option=value: unicsv/prec="6" (=default)
        cet_util: Converting from "UTF-8" to "US-ASCII", done.
        GPSBabel Version: 1.5.2
        Translation successful

But I got an empty file! With only three header columns (number, lat, lon)
even though the input GPX has elevation, time, name, comments, etc.

Am I doing something wrong? I expected a CSV file with every known field
as a separate column. Sorry for being a dunce if this is obvious...

Steve


------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
SRE
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CSV output formatting

SRE
In reply to this post by SRE
Summary: An artifact of testing caused the problem to look different...
The output you got appears unrelated to the header change, it was that
you ran a different command line than I documented in my post.

I tried your replacement header. Still no output! Then I did a hex dump:
>input bytes 0 to 255:
>  efbb bf3c 3f78 6d6c  2076 6572 7369 6f6e || <?xml version          

Turns out there was some non-printing garbage in the file. Very odd.
I copied/pasted into a fresh file, made sure there were no extra bytes.
Still no output.

So I stripped it down to just waypoints with no extensions and used a
generic header from elsewhere. Still no output! Stripped it down to
just lat/lon and name. STILL not output! Starting to feel like a
format problem instead of a file problem... something about unicsv
output not playing nice with gpx input??

gpsbabel -s -w -r -t -i gpx -f stripped.gpx -o unicsv -F stripped.csv
Translation successful
# but no waypoints! #

gpsbabel -s -w -r -t -i gpx -f stripped.gpx -o gpx -F strippedOUT.gpx
Translation successful
# got 3 waypoints #

Then it hit me that maybe the switches were the problem. The Win GUI
provided the command line above, which I really don't think about much.
Your email showed you didn't use these switches, so I tried removing them:
gpsbabel -s -w -i gpx -f s:/temp/stripped.gpx -o unicsv -F s:/temp/stripped.csv
# got 3 waypoints #
>No,Latitude,Longitude,Name
>1,49.635660,-114.587120,"TC ABC MS"
>2,56.370833,-119.924133,"TC ALCES"
>3,56.302533,-112.310000,"TC ALGARLK"

Turning route and track options off allows my ORIGINAL file to work,
even with the questionable header, but (of course?) the routes in that
file are ignored and only waypoints appear in the output CSV.

Is it a bug that adding -r and -t will inhibit waypoint output?
(checking all the boxes in the GUI doesn't seem like a bad idea)

I'm running the most recent pre-compiled release (1.5.2) on Win7.
Behavior might be different in the development version.

Steve

------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc

stripped.gpx (561 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CSV output formatting

Robert Lipe-4
Your original file had a BOM at the beginning. We can deal with that.
It had mangled xmlns entries. We can't handle that.

The interaction of -[wrt] is messy.  When GPSBabel was originally focused on serial transfers and transfers of any of those blocks could take several minutes, it made sense to have a way to read routes without pulling 10,000 trackpoints.  That's why we have them.

Some formats sense that if any of those three flags are used, act ONLY on that type of data.  Some formats ignore them completely. 

You're right, though.  unicsv is doing a somewhat dumb thing in the writer.  It's going to write ONLY one of the three types and if you specify more than one, it's going to do none because it's then not an exact match on the bitmask.  I'm not totally sure that it makes sense to merge trackpoints, waypoints, and routes all into one giant output, but doing NOTHING is not helpful.

I'll change it to add an error.

On Wed, Nov 18, 2015 at 12:18 PM, SRE <[hidden email]> wrote:
Summary: An artifact of testing caused the problem to look different...
The output you got appears unrelated to the header change, it was that
you ran a different command line than I documented in my post.

I tried your replacement header. Still no output! Then I did a hex dump:
>input bytes 0 to 255:
>  efbb bf3c 3f78 6d6c  2076 6572 7369 6f6e || <?xml version

Turns out there was some non-printing garbage in the file. Very odd.
I copied/pasted into a fresh file, made sure there were no extra bytes.
Still no output.

So I stripped it down to just waypoints with no extensions and used a
generic header from elsewhere. Still no output! Stripped it down to
just lat/lon and name. STILL not output! Starting to feel like a
format problem instead of a file problem... something about unicsv
output not playing nice with gpx input??

gpsbabel -s -w -r -t -i gpx -f stripped.gpx -o unicsv -F stripped.csv
Translation successful
# but no waypoints! #

gpsbabel -s -w -r -t -i gpx -f stripped.gpx -o gpx -F strippedOUT.gpx
Translation successful
# got 3 waypoints #

Then it hit me that maybe the switches were the problem. The Win GUI
provided the command line above, which I really don't think about much.
Your email showed you didn't use these switches, so I tried removing them:
gpsbabel -s -w -i gpx -f s:/temp/stripped.gpx -o unicsv -F s:/temp/stripped.csv
# got 3 waypoints #
>No,Latitude,Longitude,Name
>1,49.635660,-114.587120,"TC ABC MS"
>2,56.370833,-119.924133,"TC ALCES"
>3,56.302533,-112.310000,"TC ALGARLK"

Turning route and track options off allows my ORIGINAL file to work,
even with the questionable header, but (of course?) the routes in that
file are ignored and only waypoints appear in the output CSV.

Is it a bug that adding -r and -t will inhibit waypoint output?
(checking all the boxes in the GUI doesn't seem like a bad idea)

I'm running the most recent pre-compiled release (1.5.2) on Win7.
Behavior might be different in the development version.

Steve


------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: CSV output formatting

Robert Lipe-4
More thought is needed.

We probably don't want to just have unicsv ignore -[wrt] and  do it all because you'll then end up with column headers in weird places.   This isn't a problem for the original use case of sending bits over a serial hose, but is a problem here.  The GUI can't/shouldn't know about this as -[wrt] can actually change between reads and writes, can appear multiple times, and we don't want to try to allow that in the GUI by special casing unicsv with radio buttons vs. checkboxes.

The way it works today (which I'd have never guessed without looking at the code) is that it treats [wrt] in order, left to right, and the last one "wins".  Tweaking Steve's example a bit, we can see the order matters.

rjlimac7:gpsbabel robertlipe$ ./gpsbabel -s -w -t  -i gpx -f ~/Downloads/stripped.gpx -o unicsv -F -
1 3
No,Latitude,Longitude,Name
rjlimac7:gpsbabel robertlipe$ ./gpsbabel -s -t -w  -i gpx -f ~/Downloads/stripped.gpx -o unicsv -F -
2 3
No,Latitude,Longitude,Name
1,49.635660,-114.587120,"TC ABC MS"
2,56.370833,-119.924133,"TC ALCES"
3,56.302533,-112.310000,"TC ALGARLK"

I was able to add a set of checks and errors that catch the case above, but still passes the test suite. I think the GUI defaults to -w, -t , and -r "on", so the default case here may be bad but hopefully the error will lead unicsv write users to uncheck the boxes.

Committed.

On Wed, Nov 18, 2015 at 12:39 PM, Robert Lipe <[hidden email]> wrote:
Your original file had a BOM at the beginning. We can deal with that.
It had mangled xmlns entries. We can't handle that.

The interaction of -[wrt] is messy.  When GPSBabel was originally focused on serial transfers and transfers of any of those blocks could take several minutes, it made sense to have a way to read routes without pulling 10,000 trackpoints.  That's why we have them.

Some formats sense that if any of those three flags are used, act ONLY on that type of data.  Some formats ignore them completely. 

You're right, though.  unicsv is doing a somewhat dumb thing in the writer.  It's going to write ONLY one of the three types and if you specify more than one, it's going to do none because it's then not an exact match on the bitmask.  I'm not totally sure that it makes sense to merge trackpoints, waypoints, and routes all into one giant output, but doing NOTHING is not helpful.

I'll change it to add an error.

On Wed, Nov 18, 2015 at 12:18 PM, SRE <[hidden email]> wrote:
Summary: An artifact of testing caused the problem to look different...
The output you got appears unrelated to the header change, it was that
you ran a different command line than I documented in my post.

I tried your replacement header. Still no output! Then I did a hex dump:
>input bytes 0 to 255:
>  efbb bf3c 3f78 6d6c  2076 6572 7369 6f6e || <?xml version

Turns out there was some non-printing garbage in the file. Very odd.
I copied/pasted into a fresh file, made sure there were no extra bytes.
Still no output.

So I stripped it down to just waypoints with no extensions and used a
generic header from elsewhere. Still no output! Stripped it down to
just lat/lon and name. STILL not output! Starting to feel like a
format problem instead of a file problem... something about unicsv
output not playing nice with gpx input??

gpsbabel -s -w -r -t -i gpx -f stripped.gpx -o unicsv -F stripped.csv
Translation successful
# but no waypoints! #

gpsbabel -s -w -r -t -i gpx -f stripped.gpx -o gpx -F strippedOUT.gpx
Translation successful
# got 3 waypoints #

Then it hit me that maybe the switches were the problem. The Win GUI
provided the command line above, which I really don't think about much.
Your email showed you didn't use these switches, so I tried removing them:
gpsbabel -s -w -i gpx -f s:/temp/stripped.gpx -o unicsv -F s:/temp/stripped.csv
# got 3 waypoints #
>No,Latitude,Longitude,Name
>1,49.635660,-114.587120,"TC ABC MS"
>2,56.370833,-119.924133,"TC ALCES"
>3,56.302533,-112.310000,"TC ALGARLK"

Turning route and track options off allows my ORIGINAL file to work,
even with the questionable header, but (of course?) the routes in that
file are ignored and only waypoints appear in the output CSV.

Is it a bug that adding -r and -t will inhibit waypoint output?
(checking all the boxes in the GUI doesn't seem like a bad idea)

I'm running the most recent pre-compiled release (1.5.2) on Win7.
Behavior might be different in the development version.

Steve



------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Loading...