GPSBabel v1.5.3 not converting all waypoints using XCSV

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

GPSBabel v1.5.3 not converting all waypoints using XCSV

David Lee
Hi
 
I have just upgraded GPSBabel from v1.5.2 to v1.5.3 and encountered a problem where the latest version fails to convert all the waypoints in a trackpoint text file.
 
My text file contains a track comprising 6449 trackpoints. 
 
I first convert to GPX format using  GPSbabel from a command prompt using xcsv and then convert the resulting waypoint file to a track.
 
This works perfectly using GPSBabel 1.5.2 but when using v1.5.3 the output of the first step - xcsv conversion - omitted the first 5001 waypoints, returning only the last 1448. I have uninstalled/reinstalled GPSBabel several times alternating between the two versions with similar results - although with some reinstallations of v1.5.3 the first 5001 waypoints are omitted and with others 3743. V1.5.2 worked perfectly each time it was reinstalled.
 
Additionally GPSUtility will convert the original text file directly to a track with no problems.
 
 
 
My text file is in the format:
 
<Blank Line>
S DateFormat=yyyy-mm-d
S Timezone=+1:00
S Units=M,M
S SymbolSet=1
 
H R DATUM
M E               WGS 84 100  0.0000000E+00  0.0000000E+00 0 0 0
 
H  COORDINATE SYSTEM
U  LAT LON DEG
 
F Latitude   Longitude   Alt(m)   Date       Time     S
T N52.049290 W002.513730    101.7 2015-08-31 20:28:06 1
T N52.049290 W002.513780    118.1 2015-08-31 20:28:07 0
T N52.049280 W002.513780    118.1 2015-08-31 20:28:08 0
T N52.049280 W002.513780    118.0 2015-08-31 20:28:09 0
T N52.049280 W002.513780    117.9 2015-08-31 20:28:10 0
T N52.049280 W002.513780    117.8 2015-08-31 20:28:11 0
T N52.049270 W002.513780    117.6 2015-08-31 20:28:12 0
T N52.049270 W002.513780    117.4 2015-08-31 20:28:13 0
T N52.049270 W002.513780    117.1 2015-08-31 20:28:14 0
T N52.049270 W002.513780    116.9 2015-08-31 20:28:15 0
...
 
 
# gpsbabel XCSV style file
#
# Format: Anabat Express gps.txt file
# Author: David Lee
#   Date: 05/01/2016
#
 
DESCRIPTION  Convert track text file to GPX waypoints
 
# FILE LAYOUT DEFINITIIONS:
#
FIELD_DELIMITER  WHITESPACE
RECORD_DELIMITER NEWLINE
BADCHARS  SPACE
 
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
 
#
# INDIVIDUAL DATA FIELDS, IN ORDER OF APPEARANCE:
 
IFIELD IGNORE,"","%s"
 
IFIELD LAT_HUMAN_READABLE, "", "%08.5f"
IFIELD LON_HUMAN_READABLE, "", "%08.5f"
IFIELD ALT_METERS, "", "%10.0f"
IFIELD GMT_TIME,"","%Y-%m-%d"
IFIELD HMSG_TIME,"","%d:%d:%d"
 
IFIELD IGNORE,"","%s"
 
 
Command line: gpsbabel -i xcsv,style=..\gps_txt.style -f gps.txt -o gpx -F gps.gpx
 
 

------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints usingXCSV

David Lee
I should have added in my previous post that I was using Windows XP.

I have just attempted the same conversion on a Windows-7 64bit machine with
exactly the same results. ie GPSBabel v 1.5.2 will output all 6449 waypoints
but v1.5.3 returns only points 5002 to 6449.

David


----- Original Message -----
From: David Lee
To: [hidden email]
Sent: Wednesday, January 06, 2016 2:24 AM
Subject: [Gpsbabel-misc] GPSBabel v1.5.3 not converting all waypoints
usingXCSV


Hi

I have just upgraded GPSBabel from v1.5.2 to v1.5.3 and encountered a
problem where the latest version fails to convert all the waypoints in a
trackpoint text file.

My text file contains a track comprising 6449 trackpoints.

I first convert to GPX format using  GPSbabel from a command prompt using
xcsv and then convert the resulting waypoint file to a track.

This works perfectly using GPSBabel 1.5.2 but when using v1.5.3 the output
of the first step - xcsv conversion - omitted the first 5001 waypoints,
returning only the last 1448. I have uninstalled/reinstalled GPSBabel
several times alternating between the two versions with similar results -
although with some reinstallations of v1.5.3 the first 5001 waypoints are
omitted and with others 3743. V1.5.2 worked perfectly each time it was
reinstalled.

Additionally GPSUtility will convert the original text file directly to a
track with no problems.


------------------------------------------------------------------------------
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints using XCSV

tsteven4-2
In reply to this post by David Lee
Can you send your original text file so we can try to reproduce your problem?

On 1/5/2016 7:24 PM, David Lee wrote:
Hi
 
I have just upgraded GPSBabel from v1.5.2 to v1.5.3 and encountered a problem where the latest version fails to convert all the waypoints in a trackpoint text file.
 
My text file contains a track comprising 6449 trackpoints. 
 
I first convert to GPX format using  GPSbabel from a command prompt using xcsv and then convert the resulting waypoint file to a track.
 
This works perfectly using GPSBabel 1.5.2 but when using v1.5.3 the output of the first step - xcsv conversion - omitted the first 5001 waypoints, returning only the last 1448. I have uninstalled/reinstalled GPSBabel several times alternating between the two versions with similar results - although with some reinstallations of v1.5.3 the first 5001 waypoints are omitted and with others 3743. V1.5.2 worked perfectly each time it was reinstalled.
 
Additionally GPSUtility will convert the original text file directly to a track with no problems.
 
 
 
My text file is in the format:
 
<Blank Line>
S DateFormat=yyyy-mm-d
S Timezone=+1:00
S Units=M,M
S SymbolSet=1
 
H R DATUM
M E               WGS 84 100  0.0000000E+00  0.0000000E+00 0 0 0
 
H  COORDINATE SYSTEM
U  LAT LON DEG
 
F Latitude   Longitude   Alt(m)   Date       Time     S
T N52.049290 W002.513730    101.7 2015-08-31 20:28:06 1
T N52.049290 W002.513780    118.1 2015-08-31 20:28:07 0
T N52.049280 W002.513780    118.1 2015-08-31 20:28:08 0
T N52.049280 W002.513780    118.0 2015-08-31 20:28:09 0
T N52.049280 W002.513780    117.9 2015-08-31 20:28:10 0
T N52.049280 W002.513780    117.8 2015-08-31 20:28:11 0
T N52.049270 W002.513780    117.6 2015-08-31 20:28:12 0
T N52.049270 W002.513780    117.4 2015-08-31 20:28:13 0
T N52.049270 W002.513780    117.1 2015-08-31 20:28:14 0
T N52.049270 W002.513780    116.9 2015-08-31 20:28:15 0
...
 
 
# gpsbabel XCSV style file
#
# Format: Anabat Express gps.txt file
# Author: David Lee
#   Date: 05/01/2016
#
 
DESCRIPTION  Convert track text file to GPX waypoints
 
# FILE LAYOUT DEFINITIIONS:
#
FIELD_DELIMITER  WHITESPACE
RECORD_DELIMITER NEWLINE
BADCHARS  SPACE
 
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
PROLOGUE
 
#
# INDIVIDUAL DATA FIELDS, IN ORDER OF APPEARANCE:
 
IFIELD IGNORE,"","%s"
 
IFIELD LAT_HUMAN_READABLE, "", "%08.5f"
IFIELD LON_HUMAN_READABLE, "", "%08.5f"
IFIELD ALT_METERS, "", "%10.0f"
IFIELD GMT_TIME,"","%Y-%m-%d"
IFIELD HMSG_TIME,"","%d:%d:%d"
 
IFIELD IGNORE,"","%s"
 
 
Command line: gpsbabel -i xcsv,style=..\gps_txt.style -f gps.txt -o gpx -F gps.gpx
 
 


------------------------------------------------------------------------------


_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc


------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints using XCSV

David Lee
Original text file attached, together with my style file.

David


----- Original Message -----
From: tsteven4
To: David Lee ; [hidden email]
Sent: Wednesday, January 06, 2016 2:59 AM
Subject: Re: [Gpsbabel-misc] GPSBabel v1.5.3 not converting all waypoints
using XCSV


Can you send your original text file so we can try to reproduce your
problem?



------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc

gps.txt (359K) Download Attachment
gps_txt.style (873 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints using XCSV

tsteven4-2
I see 6449 points after reading.  Linux.  I am a few commits past 1.5.3.

$ ./gpsbabel -i xcsv,style=gps_txt.style -f gps.txt -x validate,debug -o gpx -F gps.gpx

Processing waypts
point ct: 6449, waypt_count: 6449

Processing routes
route head ct: 0, route_count: 0
total route point ct: 0, route_waypt_count: 0

Processing tracks
track head ct: 0, track_count: 0
total track point ct: 0, track_waypt_count: 0


On 1/5/2016 8:05 PM, David Lee wrote:
Original text file attached, together with my style file.

David


----- Original Message ----- 
From: tsteven4
To: David Lee ; [hidden email]
Sent: Wednesday, January 06, 2016 2:59 AM
Subject: Re: [Gpsbabel-misc] GPSBabel v1.5.3 not converting all waypoints 
using XCSV


Can you send your original text file so we can try to reproduce your 
problem?




------------------------------------------------------------------------------


_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc


------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints using XCSV

tsteven4-2
In reply to this post by David Lee
It also seems to work with 1.5.3 on windows 7
$ "C:/Program Files (x86)/GPSBabel/gpsbabel.exe" -i xcsv,style=gps_txt.style -f gps.txt -x validate,debug -o gpx -F gps.gpx

Processing waypts
point ct: 6449, waypt_count: 6449

Processing routes
route head ct: 0, route_count: 0
total route point ct: 0, route_waypt_count: 0

Processing tracks
track head ct: 0, track_count: 0
total track point ct: 0, track_waypt_count: 0


On 1/5/2016 8:05 PM, David Lee wrote:
Original text file attached, together with my style file.

David


----- Original Message ----- 
From: tsteven4
To: David Lee ; [hidden email]
Sent: Wednesday, January 06, 2016 2:59 AM
Subject: Re: [Gpsbabel-misc] GPSBabel v1.5.3 not converting all waypoints 
using XCSV


Can you send your original text file so we can try to reproduce your 
problem?




------------------------------------------------------------------------------


_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc


------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints using XCSV

tsteven4-2
In reply to this post by David Lee
I can reproduce an issue with the gpx writer.

On 1/5/2016 8:05 PM, David Lee wrote:
Original text file attached, together with my style file.

David


----- Original Message ----- 
From: tsteven4
To: David Lee ; [hidden email]
Sent: Wednesday, January 06, 2016 2:59 AM
Subject: Re: [Gpsbabel-misc] GPSBabel v1.5.3 not converting all waypoints 
using XCSV


Can you send your original text file so we can try to reproduce your 
problem?




------------------------------------------------------------------------------


_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc


------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints using XCSV

tsteven4-2
In reply to this post by David Lee
Here is a work around.  You may have to adjust the number upwards if you have more points or more information in each point.
./gpsbabel -i xcsv,style=gps_txt.style -f gps.txt -x validate,debug -o gpx,split=2000000 -F gps.gpx


On 1/5/2016 8:05 PM, David Lee wrote:
Original text file attached, together with my style file.

David


----- Original Message ----- 
From: tsteven4
To: David Lee ; [hidden email]
Sent: Wednesday, January 06, 2016 2:59 AM
Subject: Re: [Gpsbabel-misc] GPSBabel v1.5.3 not converting all waypoints 
using XCSV


Can you send your original text file so we can try to reproduce your 
problem?




------------------------------------------------------------------------------


_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc


------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints using XCSV

tsteven4-2
In reply to this post by David Lee
I think this was introduced in
https://github.com/gpsbabel/gpsbabel/commit/e529697c7247b0df99c94e05c73045efb2ed5999

Hack up GPX writer to prepare for GGZ.

optionally split output in gpx writer.
Keep a list of files we’ve created so that GGZ writer can turn around
and read them.


On 1/5/2016 8:05 PM, David Lee wrote:
Original text file attached, together with my style file.

David


----- Original Message ----- 
From: tsteven4
To: David Lee ; [hidden email]
Sent: Wednesday, January 06, 2016 2:59 AM
Subject: Re: [Gpsbabel-misc] GPSBabel v1.5.3 not converting all waypoints 
using XCSV


Can you send your original text file so we can try to reproduce your 
problem?




------------------------------------------------------------------------------


_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc


------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints using XCSV

Robert Lipe-4
Ugh. That was indeed mine. I clearly submitted an incomplete change that sabotaged our GPX writer.  The cone of shame is mine for sabotaging our 1.5.3 release. :-/
I didn't even submit enough of ggz support to enable it for this release as I ran into internationalization issues that, ironically, Qt made worse for us.  ggz has to call the gpx writer to write a subset of the output as individual gpx files.  It then turns around and reads the gpx files and then writes an index that contains byte (not character) offsets of the opening and <wpt> and the length.  split was to be the approximate byte count of the destination.  The 'split' option was meant to be a semi-secret handshake between ggz and gpx to keep some semblance of separation. Unfortunately, I had pieces of this source in development on four different computers over the course of several months and the version I submitted screwed up the 'optional' part of that comment.

Let's let 1.5.3 simmer a bit. That's one person out of about 1500 that's noticed. (I get that NOT noticing when we silently drop data in GPX is really bad...)  If there are other fixes needed, I'll bump to a 1.5.4 (with all the overhead that entails) but if this is the biggest source of egg on my face in a week or two, I'm more tempted to just comment out that block that closes and reopens the gpx output and silently re-reroll the 1.5.3 as a slipstream release.

I already know that some in the audience find that practice distasteful (I'm not thrilled with it....)  but the overhead of a version bump is a costly undertaking for me.

RJL


On Tue, Jan 5, 2016 at 9:43 PM, tsteven4 <[hidden email]> wrote:
I think this was introduced in
https://github.com/gpsbabel/gpsbabel/commit/e529697c7247b0df99c94e05c73045efb2ed5999

Hack up GPX writer to prepare for GGZ.

optionally split output in gpx writer.
Keep a list of files we’ve created so that GGZ writer can turn around
and read them.


On 1/5/2016 8:05 PM, David Lee wrote:
Original text file attached, together with my style file.

David


----- Original Message ----- 
From: tsteven4
To: David Lee ; [hidden email]
Sent: Wednesday, January 06, 2016 2:59 AM
Subject: Re: [Gpsbabel-misc] GPSBabel v1.5.3 not converting all waypoints 
using XCSV


Can you send your original text file so we can try to reproduce your 
problem?




------------------------------------------------------------------------------


_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc



------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints using XCSV

Mat-53
OMG .. he is human after all! ;-)

I dont have much to say here, but I follow this misc list and I was
almost convinced RJL was a script running in some dark basement.
I applaud your candor and got a laugh from your 'slipstream' approach.
Good job :-)

Mat

On 2016-01-06 16:23, Robert Lipe wrote:

> Ugh. That was indeed mine. I clearly submitted an incomplete change
> that sabotaged our GPX writer.  The cone of shame is mine for
> sabotaging our 1.5.3 release. :-/
>
> I didn't even submit enough of ggz support to enable it for this
> release as I ran into internationalization issues that, ironically, Qt
> made worse for us.  ggz has to call the gpx writer to write a subset
> of the output as individual gpx files.  It then turns around and reads
> the gpx files and then writes an index that contains byte (not
> character) offsets of the opening and <wpt> and the length.  split was
> to be the approximate byte count of the destination.  The 'split'
> option was meant to be a semi-secret handshake between ggz and gpx to
> keep some semblance of separation. Unfortunately, I had pieces of this
> source in development on four different computers over the course of
> several months and the version I submitted screwed up the 'optional'
> part of that comment.
>
> Let's let 1.5.3 simmer a bit. That's one person out of about 1500
> that's noticed. (I get that NOT noticing when we silently drop data in
> GPX is really bad...)  If there are other fixes needed, I'll bump to a
> 1.5.4 (with all the overhead that entails) but if this is the biggest
> source of egg on my face in a week or two, I'm more tempted to just
> comment out that block that closes and reopens the gpx output and
> silently re-reroll the 1.5.3 as a slipstream release.
>
> I already know that some in the audience find that practice
> distasteful (I'm not thrilled with it....)  but the overhead of a
> version bump is a costly undertaking for me.
>
> RJL
>
> On Tue, Jan 5, 2016 at 9:43 PM, tsteven4 <[hidden email]> wrote:
>
>> I think this was introduced in
>>
>>>
>>
> https://github.com/gpsbabel/gpsbabel/commit/e529697c7247b0df99c94e05c73045efb2ed5999
>>> [1]
>>
>>> Hack up GPX writer to prepare for GGZ.
>>>
>>> optionally split output in gpx writer.
>>> Keep a list of files we’ve created so that GGZ writer can turn
>>> around
>>> and read them.
>>
>> On 1/5/2016 8:05 PM, David Lee wrote:
>>
>>> Original text file attached, together with my style file.
>>>
>>> David
>>>
>>> ----- Original Message -----
>>> From: tsteven4
>>> To: David Lee ; [hidden email]
>>> Sent: Wednesday, January 06, 2016 2:59 AM
>>> Subject: Re: [Gpsbabel-misc] GPSBabel v1.5.3 not converting all
>>> waypoints
>>> using XCSV
>>>
>>> Can you send your original text file so we can try to reproduce
>>> your
>>> problem?
>>>
>>>
>>
> ------------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Gpsbabel-misc mailing list http://www.gpsbabel.org [2]
>>> [hidden email]
>>> To unsubscribe, change list options, or see archives, visit:
>>> https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc [3]
>
>
>
> Links:
> ------
> [1]
> https://github.com/gpsbabel/gpsbabel/commit/e529697c7247b0df99c94e05c73045efb2ed5999
> [2] http://www.gpsbabel.org
> [3] https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Gpsbabel-misc mailing list http://www.gpsbabel.org
> [hidden email]
> To unsubscribe, change list options, or see archives, visit:
> https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc


------------------------------------------------------------------------------
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints using XCSV

Robert Lipe-4


On Tue, Jan 5, 2016 at 11:45 PM, Mat <[hidden email]> wrote:
OMG .. he is human after all! ;-)

It's true.  Even down to the 'computers don't make mistakes. humans make mistakes.' aspect, I'm human.
 
I dont have much to say here, but I follow this misc list and I was
almost convinced RJL was a script running in some dark basement.

In the 15-ish years since I've created this code, I've run off and on in cycles of high demand. Whether that's health, day job, family, or whatever, GPSBabel is rarely the only thing tugging on my pants leg at any given moment. December, for example, involved a trip to Brazil to train indigenous people on ethnographic mapping to preserve their cultures, a few days hiking in a state park in Sao Paulo, buying a new house, and a bit of vacation. I'm often typing from a phone or tablet in bed or on the go or at least with a dozen other things demanding my attention.  I can sound mechanical and terse, so I don't blame you for assuming I'm not Turing-complete. I'd like to be able to hand-write a thank you note (though nobody could read it...my handwriting is awful) to all the 20M-ish users of or contributors to GPSBabel, but I'm aware that I sound a bit like a robot sometimes. I'd rather fire off a one-word "committed" or "thanx" than either nothing or deadlocking on me buying a quill, cotton stationery,and postage.

Steve has absolutely stepped up in recent years to help keep GPSBabel going and he totally diagnosed my screwup on this one.  Thank you, Steve.


So the best I can say is that I'm in a well-lit bonus room and not a dark basement. :-)  Robert Lipe is an actual human.
 
I applaud your candor and got a laugh from your 'slipstream' approach.
Good job :-)

Thanx. I screw up.  I try to find repairs for those mistakes that won't make any of us (well, at least most of us...) totally crazy. I'm not always proud of the compromises involved.  This is exactly why I try to "think out loud" from time to time to explain how we got where we are.

You don't have to be creative to hear the profanity I uttered as I read that thread unfolding. The minute Steve uttered that it was in the GPX writer, I was like "oh, but the GPX writer hasn't changed other than... CRAP!"

Your humble Chief Babel-Head,
Robert (a.k.a. RJL)



 

Mat

On 2016-01-06 16:23, Robert Lipe wrote:
> Ugh. That was indeed mine. I clearly submitted an incomplete change
> that sabotaged our GPX writer.  The cone of shame is mine for
> sabotaging our 1.5.3 release. :-/
>
> I didn't even submit enough of ggz support to enable it for this
> release as I ran into internationalization issues that, ironically, Qt
> made worse for us.  ggz has to call the gpx writer to write a subset
> of the output as individual gpx files.  It then turns around and reads
> the gpx files and then writes an index that contains byte (not
> character) offsets of the opening and <wpt> and the length.  split was
> to be the approximate byte count of the destination.  The 'split'
> option was meant to be a semi-secret handshake between ggz and gpx to
> keep some semblance of separation. Unfortunately, I had pieces of this
> source in development on four different computers over the course of
> several months and the version I submitted screwed up the 'optional'
> part of that comment.
>
> Let's let 1.5.3 simmer a bit. That's one person out of about 1500
> that's noticed. (I get that NOT noticing when we silently drop data in
> GPX is really bad...)  If there are other fixes needed, I'll bump to a
> 1.5.4 (with all the overhead that entails) but if this is the biggest
> source of egg on my face in a week or two, I'm more tempted to just
> comment out that block that closes and reopens the gpx output and
> silently re-reroll the 1.5.3 as a slipstream release.
>
> I already know that some in the audience find that practice
> distasteful (I'm not thrilled with it....)  but the overhead of a
> version bump is a costly undertaking for me.
>
> RJL
>
> On Tue, Jan 5, 2016 at 9:43 PM, tsteven4 <[hidden email]> wrote:
>
>> I think this was introduced in
>>
>>>
>>
> https://github.com/gpsbabel/gpsbabel/commit/e529697c7247b0df99c94e05c73045efb2ed5999
>>> [1]
>>
>>> Hack up GPX writer to prepare for GGZ.
>>>
>>> optionally split output in gpx writer.
>>> Keep a list of files we’ve created so that GGZ writer can turn
>>> around
>>> and read them.
>>
>> On 1/5/2016 8:05 PM, David Lee wrote:
>>
>>> Original text file attached, together with my style file.
>>>
>>> David
>>>
>>> ----- Original Message -----
>>> From: tsteven4
>>> To: David Lee ; [hidden email]
>>> Sent: Wednesday, January 06, 2016 2:59 AM
>>> Subject: Re: [Gpsbabel-misc] GPSBabel v1.5.3 not converting all
>>> waypoints
>>> using XCSV
>>>
>>> Can you send your original text file so we can try to reproduce
>>> your
>>> problem?
>>>
>>>
>>
> ------------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Gpsbabel-misc mailing list http://www.gpsbabel.org [2]
>>> [hidden email]
>>> To unsubscribe, change list options, or see archives, visit:
>>> https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc [3]
>
>
>
> Links:
> ------
> [1]
> https://github.com/gpsbabel/gpsbabel/commit/e529697c7247b0df99c94e05c73045efb2ed5999
> [2] http://www.gpsbabel.org
> [3] https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Gpsbabel-misc mailing list http://www.gpsbabel.org
> [hidden email]
> To unsubscribe, change list options, or see archives, visit:
> https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc


------------------------------------------------------------------------------
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc


------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints using XCSV

David Lee
In reply to this post by Robert Lipe-4

Thanks for the rapid replies guys.
 
tsteven4's suggested workaround appears to be perfectly effective so no urgency to provide a "proper" fix. However it would be useful to add a notification about this workaround on the GBSBabel website.
 
The text files I am dealing with are produced by a commercial bat detector with built in GPS - the Anabat Express. Whilst the file format was designed to be compatible with GPS Utility there are great advantages in using GBPBabel instead - particularly since it allows the extraction of additional information, including species data, from files not originally intended for GPS data extraction.
 
Would it be possible to include XCSV files as a data source in the GPSBabel GUI? UK batworkers tend not to be very computer literate and are likely to have difficulties with console-mode applications!
 
Regards
 
David Lee
 
 
 
----- Original Message -----
Sent: Wednesday, January 06, 2016 5:23 AM
Subject: Re: [Gpsbabel-misc] GPSBabel v1.5.3 not converting all waypoints using XCSV

Ugh. That was indeed mine. I clearly submitted an incomplete change that sabotaged our GPX writer.  The cone of shame is mine for sabotaging our 1.5.3 release. :-/
I didn't even submit enough of ggz support to enable it for this release as I ran into internationalization issues that, ironically, Qt made worse for us.  ggz has to call the gpx writer to write a subset of the output as individual gpx files.  It then turns around and reads the gpx files and then writes an index that contains byte (not character) offsets of the opening and <wpt> and the length.  split was to be the approximate byte count of the destination.  The 'split' option was meant to be a semi-secret handshake between ggz and gpx to keep some semblance of separation. Unfortunately, I had pieces of this source in development on four different computers over the course of several months and the version I submitted screwed up the 'optional' part of that comment.

Let's let 1.5.3 simmer a bit. That's one person out of about 1500 that's noticed. (I get that NOT noticing when we silently drop data in GPX is really bad...)  If there are other fixes needed, I'll bump to a 1.5.4 (with all the overhead that entails) but if this is the biggest source of egg on my face in a week or two, I'm more tempted to just comment out that block that closes and reopens the gpx output and silently re-reroll the 1.5.3 as a slipstream release.

I already know that some in the audience find that practice distasteful (I'm not thrilled with it....)  but the overhead of a version bump is a costly undertaking for me.

RJL

------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints using XCSV

Robert Lipe-4
Looking at that CL, I agree that rolling it back is the thing to do.  I'll do that in a few minutes.  I'll wait a few days before respinning to see if there's anything else that pops up. I'll probably retag as 1.5.3a. Assuming nothing radical is going on in the trunk I'll probably respin from HEAD and not branch the actual release. You own the EXIF issue and that path is so little used, you can get radical there if you see fit.

David, we don't include xcsv in the GUI because it's not really usable as a pointy-clicky option; it requires creation of a style file which doesn't lend itself well to GUI action. 

What we prefer instead is to just include user-contributed style files.  If you can send your style definition and a few paragraphs describing it like we have in http://www.gpsbabel.org/htmldoc-1.5.3/The_Formats.html, I'll drop it into the respin.  Then it'll be in the GUI, automatically tested and available to the next user of Anabat. If you can do the docbook, great, but it's more important for you to provide the words describing how to use it, what eversions of Anabat it works with, etc. than to fret about formatting.

For the inevitable "but wait I asked for a format and you told me to get lost" response, our style definitions are very lightweight compared to a custom C++ module. 

On Wed, Jan 6, 2016 at 11:27 AM, David Lee <[hidden email]> wrote:
Thanks for the rapid replies guys.
 
tsteven4's suggested workaround appears to be perfectly effective so no urgency to provide a "proper" fix. However it would be useful to add a notification about this workaround on the GBSBabel website.
 
The text files I am dealing with are produced by a commercial bat detector with built in GPS - the Anabat Express. Whilst the file format was designed to be compatible with GPS Utility there are great advantages in using GBPBabel instead - particularly since it allows the extraction of additional information, including species data, from files not originally intended for GPS data extraction.
 
Would it be possible to include XCSV files as a data source in the GPSBabel GUI? UK batworkers tend not to be very computer literate and are likely to have difficulties with console-mode applications!
 
Regards
 
David Lee
 
 
 
----- Original Message -----
Sent: Wednesday, January 06, 2016 5:23 AM
Subject: Re: [Gpsbabel-misc] GPSBabel v1.5.3 not converting all waypoints using XCSV

Ugh. That was indeed mine. I clearly submitted an incomplete change that sabotaged our GPX writer.  The cone of shame is mine for sabotaging our 1.5.3 release. :-/
I didn't even submit enough of ggz support to enable it for this release as I ran into internationalization issues that, ironically, Qt made worse for us.  ggz has to call the gpx writer to write a subset of the output as individual gpx files.  It then turns around and reads the gpx files and then writes an index that contains byte (not character) offsets of the opening and <wpt> and the length.  split was to be the approximate byte count of the destination.  The 'split' option was meant to be a semi-secret handshake between ggz and gpx to keep some semblance of separation. Unfortunately, I had pieces of this source in development on four different computers over the course of several months and the version I submitted screwed up the 'optional' part of that comment.

Let's let 1.5.3 simmer a bit. That's one person out of about 1500 that's noticed. (I get that NOT noticing when we silently drop data in GPX is really bad...)  If there are other fixes needed, I'll bump to a 1.5.4 (with all the overhead that entails) but if this is the biggest source of egg on my face in a week or two, I'm more tempted to just comment out that block that closes and reopens the gpx output and silently re-reroll the 1.5.3 as a slipstream release.

I already know that some in the audience find that practice distasteful (I'm not thrilled with it....)  but the overhead of a version bump is a costly undertaking for me.

RJL


------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
SRE
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints using XCSV

SRE
At 01:59 PM 1/6/2016, Robert Lipe wrote:
>David, we don't include xcsv in the GUI because it's not really usable as a pointy-clicky option;
>it requires creation of a style file which doesn't lend itself well to GUI action.

One could provide a pathname input box for the style file when the xcsv option was chosen...
building the style file would still be manual, but using it would be GUI-friendly and would
not require so many pseudo-formats.

I admit being very confused by all the format options and am only now (and perhaps incorrectly)
coming to realize some (many?) of them are the same format with a different style file. Maybe
choose xcsv, pull down a list of pre-defined style files or choose "I have my own" and get a
pathname entry box?? I expected the current CSV format option to read generic CSV data,
whatever was there, so adding "(use xcsv for generic CSV input)" might be helpful.

Steve


------------------------------------------------------------------------------
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GPSBabel v1.5.3 not converting all waypoints using XCSV

Robert Lipe-4


On Wed, Jan 6, 2016 at 6:25 PM, SRE <[hidden email]> wrote:
At 01:59 PM 1/6/2016, Robert Lipe wrote:
>David, we don't include xcsv in the GUI because it's not really usable as a pointy-clicky option;
>it requires creation of a style file which doesn't lend itself well to GUI action.

One could provide a pathname input box for the style file when the xcsv option was chosen...
building the style file would still be manual, but using it would be GUI-friendly

If the GUI actually  had knowledge of specific the formats and flags (and it's easy to argue it should, but it doesn't) that would be the way to do it.  The GUI asks the command line version for a list of formats and flags and gets a description that says "this argument takes an integer between 9 and 15" (whatever) so it displays a dropdown with those options or a checkbox for the boolean options or whatever. Very, very few of our options accept a filename, so it's expressed as a string. The GUI doesn't know to launch a file picker. Could it be special cased? Yes. It just needs a programmer that wants it badly enough to make it happen.  
 
and would not require so many pseudo-formats.

They're not pseudo-formats. While we provide a simplified internal way to describe them, they are distinct to the user. It's not like geonet and mxf have any more in common to a user than Garmin and TomTom POI files do.

I do regret naming the Delorme format 'csv', but it was like the third or fourth format I supported so it made sense at the time and I certainly wasn't thinking our list would ever grow way past a hundred formats. I flirted with the idea of renaming it, but I sense there are so many scripts that use the 'csv' format for just schlepping around points that would blow up on us, too.
 
I admit being very confused by all the format options and am only now (and perhaps incorrectly)
coming to realize some (many?) of them are the same format with a different style file. Maybe

That dropdown is admittedly out of control.  I did some work years ago to allow people turn off the formats they don't care abut in settings, but I didn't take it far enough. The reality is that the top dozen[1] or so of our formats so totally dominate the usage of the others in the GUI, that MOST of them should default 'off' with an option given 'don't see your format here? see more' that presented the user with a searchable, progressive disclosure interface that didn't overwhelm them with all 160-ish formats.

Our GUI hasn't received much attention for a Long Time.  One of the last big investments I made in it was getting translations plumbed through. I thought that would be huge for our non-English parts of the world, but translator participation has been at zero and it's been years since we've had new languages or even requests.

 
choose xcsv, pull down a list of pre-defined style files or choose "I have my own" and get a
pathname entry box?? I expected the current CSV format option to read generic CSV data,
whatever was there, so adding "(use xcsv for generic CSV input)" might be helpful.

We recently strengthened those words in the doc. Contrast
 

[1] I just looked at the use frequency in 2015. The 10'th most popular format is used 4% as often as #1 and there's a long tail down from there to #176 on the list, palmdoc. 

------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Loading...