Waypoint name with non-ascii incorrecto on Garmin

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Waypoint name with non-ascii incorrecto on Garmin

Göran Uddeborg
Hello,

With gpsbabel 1.4 I specified the character set when sending waypoints
(geocaches) to my Garmin Etrex Legend HCx.  After I upgraded to 1.5,
gpsbabel no longer accepts any "-c" option, and character set
selection is supposed to be done automatically.  It seems to work fine
for the note section, but then name of the waypoint gets mangled.

I've created a "loc" file to illustrate the problem.  It uses the
letters a-ring, a-diaeresis, and o-diaeresis both in the name and the
note.  More exactly, both are "A å-ä-ö more".  I load it to my GPS
using the command

    gpsbabel -i geo -f test.loc -o garmin -F /dev/ttyUSB0

When I look at it in my GPS, the note looks right.  But the name has
become "A ï-ï-ï".  That is, all the non-ascii letters have been
replaced with i-diaeresis.  And the "more" part has been cut off,
despite there being place for it in the GPS.

Do I do something wrong?  Is this a bug in gpsbabel?  If the latter,
is there a way to work around it?

- This is on a Fedora system
- I'm using the gpsbabel-1.5.2-1.fc22.x86_64.
- The loc file is in UTF-8.
- My locale is sv_SE.utf8.


<loc version="1.0" src="Groundspeak">
<waypoint>
        <name id="A å-ä-ö more">A å-ä-ö more</name>
<coord lat="60" lon="10"/>
<type>Geocache</type>
</waypoint></loc>

------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|

Re: Waypoint name with non-ascii incorrecto on Garmin

Robert Lipe-4
Thanx for the good description.

This is a sticky one.  Garmin protocol says everything shall be ascii-only - worse, still, it says upper case letters and digits only.  We whitelist some models - including yours - that are known to not burst into flames when they see diacriticals or the horror of a lower case letter.  When we moved our internals to use a consistently encoded data type, it looks like we may have mishandled receiver_charset in garmin.cc.  My suspicion is that we need to get the Windows 1250 codec http://doc.qt.io/qt-4.8/qtextcodec.html for the whitelisted devices and then call fromUnicode() for each of the strings (name, comment, description) that we can write to the device and probably do the same when reading from the device.

Would you like to take a swing at that?

On Sun, Aug 23, 2015 at 9:22 AM, Göran Uddeborg <[hidden email]> wrote:
Hello,

With gpsbabel 1.4 I specified the character set when sending waypoints
(geocaches) to my Garmin Etrex Legend HCx.  After I upgraded to 1.5,
gpsbabel no longer accepts any "-c" option, and character set
selection is supposed to be done automatically.  It seems to work fine
for the note section, but then name of the waypoint gets mangled.

I've created a "loc" file to illustrate the problem.  It uses the
letters a-ring, a-diaeresis, and o-diaeresis both in the name and the
note.  More exactly, both are "A å-ä-ö more".  I load it to my GPS
using the command

    gpsbabel -i geo -f test.loc -o garmin -F /dev/ttyUSB0

When I look at it in my GPS, the note looks right.  But the name has
become "A ï-ï-ï".  That is, all the non-ascii letters have been
replaced with i-diaeresis.  And the "more" part has been cut off,
despite there being place for it in the GPS.

Do I do something wrong?  Is this a bug in gpsbabel?  If the latter,
is there a way to work around it?

- This is on a Fedora system
- I'm using the gpsbabel-1.5.2-1.fc22.x86_64.
- The loc file is in UTF-8.
- My locale is sv_SE.utf8.


<loc version="1.0" src="Groundspeak">
<waypoint>
        <name id="A å-ä-ö more">A å-ä-ö more</name>
<coord lat="60" lon="10"/>
<type>Geocache</type>
</waypoint></loc>

------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc



------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|

Re: Waypoint name with non-ascii incorrecto on Garmin

Göran Uddeborg
Robert Lipe:
> Garmin protocol says everything shall be ascii-only -
> worse, still, it says upper case letters and digits only.

It does! Weird, since it supports a lot of non-ASCII letters from the
manual interface.  Both upper and lower case.

> My suspicion is that we need to get
> the Windows 1250 codec http://doc.qt.io/qt-4.8/qtextcodec.html for the
> whitelisted devices and then call fromUnicode() for each of the strings (name,
> comment, description) that we can write to the device and probably do the same
> when reading from the device.
>
> Would you like to take a swing at that?

Given your description, it sounds like it wouldn't be too hard to do.
(Assuming the code in structured enough so even I can find my way.:-))
I'll try to get some time to have a look at this.

------------------------------------------------------------------------------
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|

Re: Waypoint name with non-ascii incorrecto on Garmin

Robert Lipe-4


On Tue, Aug 25, 2015 at 8:42 AM, Göran Uddeborg <[hidden email]> wrote:
Robert Lipe:
> Garmin protocol says everything shall be ascii-only -
> worse, still, it says upper case letters and digits only.

It does! Weird, since it supports a lot of non-ASCII letters from the
manual interface.  Both upper and lower case.

What the specs say and what any individual device actually does are only loosely related.

Older Garmins, for example, will actually crash if a space appears in a waypoint name.

 
> My suspicion is that we need to get
> the Windows 1250 codec http://doc.qt.io/qt-4.8/qtextcodec.html for the
> whitelisted devices and then call fromUnicode() for each of the strings (name,
> comment, description) that we can write to the device and probably do the same
> when reading from the device.
>
> Would you like to take a swing at that?

Given your description, it sounds like it wouldn't be too hard to do.
(Assuming the code in structured enough so even I can find my way.:-))
I'll try to get some time to have a look at this.


The code in question is in garmin.cc in waypoint_prepare.  It's kind of an ugly collision between QStrings (a sensible, modern, string class that has a consistent internal encoding) and C strings (everything is implicitly UTF-8) and fixed length buffers that go over the wire.

My starting place would be just before the call to mkshort().  There, src is a QString and we're about to create the C string.  That's when we need to decompose from Unicode to MS_ANSI for your device.

Then, in rw_init() depending whether receiver_charset = CET_CHARSET_MS_ANSI, create a codec for either Windows 1250 (MS ANSI) or UTF-8 and use that above.

Waypoint read and (track|route)(read|write) would need similar treatment, though they easily see an order less magnitude of use than waypoint write for us. Perhaps there's a more central point.

Steve's probably the only other one in the list with experience in handling codecs in this way, so if he contradicts anything above, he's probably right.  (Steve, this used to work because this code was calling cet_convert_init() with the alternate charset and letting CET do it.)


I'm trying to get more people involved in the code...

RJL

------------------------------------------------------------------------------

_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|

Re: Waypoint name with non-ascii incorrect on Garmin

Göran Uddeborg
It took me quite a while to get back to this.  But better late than
never!  I include the latest mail in extenso so you don't have to dig
in the archives.

Robert Lipe:

> On Tue, Aug 25, 2015 at 8:42 AM, Göran Uddeborg <[hidden email]> wrote:
>
>     Robert Lipe:
>     > Garmin protocol says everything shall be ascii-only -
>     > worse, still, it says upper case letters and digits only.
>    
>     It does! Weird, since it supports a lot of non-ASCII letters from the
>     manual interface.  Both upper and lower case.
>
> What the specs say and what any individual device actually does are only
> loosely related.
>
> Older Garmins, for example, will actually crash if a space appears in a
> waypoint name.
>
>  
>
>     > My suspicion is that we need to get
>     > the Windows 1250 codec http://doc.qt.io/qt-4.8/qtextcodec.html for the
>     > whitelisted devices and then call fromUnicode() for each of the strings
>     (name,
>     > comment, description) that we can write to the device and probably do the
>     same
>     > when reading from the device.
>     >
>     > Would you like to take a swing at that?
>    
>     Given your description, it sounds like it wouldn't be too hard to do.
>     (Assuming the code in structured enough so even I can find my way.:-))
>     I'll try to get some time to have a look at this.
>
> The code in question is in garmin.cc in waypoint_prepare.  It's kind of an ugly
> collision between QStrings (a sensible, modern, string class that has a
> consistent internal encoding) and C strings (everything is implicitly UTF-8)
> and fixed length buffers that go over the wire.
>
> My starting place would be just before the call to mkshort().  There, src is a
> QString and we're about to create the C string.  That's when we need to
> decompose from Unicode to MS_ANSI for your device.
>
> Then, in rw_init() depending whether receiver_charset = CET_CHARSET_MS_ANSI,
> create a codec for either Windows 1250 (MS ANSI) or UTF-8 and use that above.
>
> Waypoint read and (track|route)(read|write) would need similar treatment,
> though they easily see an order less magnitude of use than waypoint write for
> us. Perhaps there's a more central point.
>
> Steve's probably the only other one in the list with experience in handling
> codecs in this way, so if he contradicts anything above, he's probably right.  
> (Steve, this used to work because this code was calling cet_convert_init() with
> the alternate charset and letting CET do it.)
>
> I'm trying to get more people involved in the code...

The usage of QStrings and char*-strings was indeed a bit tricky to
follow occasionally.  I have found a patch that seems to do the right
thing, but I'm not sure I follow your intentions.  So let me explain
how I understood it.

- waypoint_prepare in garmin.cc gets called with a waypoint where the
  strings have been converted to "MS_ANSI".

- waypoint_prepare calls mkshort() with a char*-string second
  argument.

- mkshort() notices that the string is not UTF-8 (hdl->is_utf8 is
  false).  So it calls xxstrdup() to duplicate the string.

- xxstrdup() is #define'd to xstrdup().

- There are two overloaded versions of xstrdup util.cc, xstrdup(const
  char* s) and xstrdup(const QString& s).  However, only the second is
  declared in defs.h, so only the second is a candidate when compiling
  mkshort.cc.

- Because of that, the compiler will insert an implicit conversion of
  the char*-string to QString.  This conversion assumes the
  char*-string is in UTF-8, which it isn't; this is where the string
  gets mangled.

>From what I understand, there is no reason to go via xstrdup(const
QString&) in this xstrdup call.  It would make more sence to call
xstrdup(const char*) directly.

So I added a declaration of xstrdup(const char* s) to defs.h.  (In the
branch where DEBUG_MEM is undefined.  If DEBUG_MEM is defined, the
function is actually called XSTRDUP, distinct from the QString case. I
believe the problem wouldn't have appeared with that definition,
though I haven't verified it.)  This makes the call from mkshort go
directly to xstrdup(const char*), without any mangling of the name.
And when I look in my GPS, it looks correct. :-)

Is this a correct understanding, or am I missing something?  The
trivial patch I applied is attached below.

diff --git a/defs.h b/defs.h
index 9a8cf03..7c9a39f 100644
--- a/defs.h
+++ b/defs.h
@@ -902,6 +902,7 @@ char* xstrappend(char* src, const char* addon);
 #define xxrealloc(p, s, file, line) xrealloc(p,s)
 #define xxfree(mem, file, line) xfree(mem)
 #define xxstrdup(s, file, line) xstrdup(s)
+char *xstrdup(const char* s);
 #define xxstrappend(src, addon, file, line) xstrappend(src, addon)
 #else /* DEBUG_MEM */
 void* XCALLOC(size_t nmemb, size_t size, DEBUG_PARAMS);

------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|

Re: Waypoint name with non-ascii incorrect on Garmin

Göran Uddeborg
> The trivial patch I applied is attached below.

Realising I'm old-fashioned sending patches in e-mail, I've now made a
pull request on github.  That's how it's supposed to be done these
days I figure. :-)

------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|

Re: Waypoint name with non-ascii incorrect on Garmin

Robert Lipe-4
I'm not ignoring you.  Last week was a US holiday and I was on the road. I'm leaving in the morning for two weeks of international travel.

Thank you for the great explanation and deep dive.  This seems like it's probably right.  That code is a messy intersection of encodings and string representations.

On Mon, Nov 30, 2015 at 11:25 AM, Göran Uddeborg <[hidden email]> wrote:
> The trivial patch I applied is attached below.

Realising I'm old-fashioned sending patches in e-mail, I've now made a
pull request on github.  That's how it's supposed to be done these
days I figure. :-)


------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc
Reply | Threaded
Open this post in threaded view
|

Re: Waypoint name with non-ascii incorrect on Garmin

Göran Uddeborg
Robert Lipe:
> I'm not ignoring you.

I didn't mean to imply that.  I just wanted to show that I too could
play the way kids play today! :-)

------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140
_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
[hidden email]
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc