Garmin GPS database reader

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Garmin GPS database reader

Olaf Klein
Dear all,

this is my first try to support newer versions of mapsource, now called Garmin
GPS database. Because this new database version is a follower of MapSource,
the question was, put more code for dependences in mapsource or start a new
way. I took the new way to learn more about mapsource internals and establish
a new basis for upcomming MapSource releases. Of cource the basic for the new
way was mapsource from Robert.

Most problems i had was the sequence of route points. I've found no
connections with real data and MapSource versions and any unknown bytes
within data. I've tested the resulting code against my private gdb collection
of nearly 200 files. These files was created by MapSource V6.0 to current
beta releases.

O.K.



019-gdb.patch (21K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Lowrance Patch

rnlnero
Added support for routes, icons, and tracks.  Icons are by default converted to
waypoints.  Use the -ignoreicons option to not convert them to waypoints.  It
hsa been tested with the iFinder H2O.

Reference cases (*.usr) should be put in the reference directory.
(Sourceforge.net rejected a zip compilation.) Notice the patch for the
reference gpx files, accounting for the standardized precision in lat/long
coordinate display.

Ling
--


lowrance.patch (60K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Lowrance Patch

Robert Lipe
Russ & Ling wrote:

> Added support for routes, icons, and tracks.  Icons are by default
> converted to waypoints.  Use the -ignoreicons option to not convert

I've committed this, but it does fail for me.  (No, I'm not happy about
having two formats checked in that are known to fail, but I hope that by
having more balls int he air at the same time we can work these things
out a little faster.)  Please investigate the following failure.


+ ./gpsbabel -i geo -f geocaching.loc -o lowranceusr -F /tmp/gpsbabel.7545/lowrance1.usr
+ bincompare /tmp/gpsbabel.7545/lowrance1.usr reference/lowrance.usr
+ rm -f /tmp/gpsbabel.7545/bc1
+ rm -f /tmp/gpsbabel.7545/bc2
+ od -Ax -txC -v /tmp/gpsbabel.7545/lowrance1.usr
+ od -Ax -txC -v reference/lowrance.usr
+ diff -u /tmp/gpsbabel.7545/bc1 /tmp/gpsbabel.7545/bc2
--- /tmp/gpsbabel.7545/bc1      2005-06-27 14:00:14.000000000 -0500
+++ /tmp/gpsbabel.7545/bc2      2005-06-27 14:00:14.000000000 -0500
@@ -1,24 +1,24 @@
 000000 02 00 00 00 09 00 00 00 ff 57 41 00 3c 7d 6c ff
 000010 00 00 00 00 05 00 00 00 47 43 45 42 42 00 00 00
-000020 00 d0 89 6d 38 36 27 00 00 00 00 01 00 95 97 41
+000020 00 c0 7b 6d 38 36 27 00 00 00 00 01 00 95 97 41
 000030 00 7d 42 6d ff 00 00 00 00 06 00 00 00 47 43 31
-000040 41 33 37 00 00 00 00 d0 89 6d 38 36 27 00 00 00
+000040 41 33 37 00 00 00 00 c0 7b 6d 38 36 27 00 00 00
 000050 00 02 00 fa 64 41 00 3f 5c 6d ff 00 00 00 00 06
-000060 00 00 00 47 43 31 43 32 42 00 00 00 00 d0 89 6d
+000060 00 00 00 47 43 31 43 32 42 00 00 00 00 c0 7b 6d
 000070 38 36 27 00 00 00 00 03 00 98 7b 41 00 e5 4f 6d
 000080 ff 00 00 00 00 06 00 00 00 47 43 32 35 41 39 00
-000090 00 00 00 d0 89 6d 38 36 27 00 00 00 00 04 00 1d
+000090 00 00 00 c0 7b 6d 38 36 27 00 00 00 00 04 00 1d
 0000a0 a3 41 00 86 27 6d ff 00 00 00 00 06 00 00 00 47
-0000b0 43 32 37 32 33 00 00 00 00 d0 89 6d 38 36 27 00
+0000b0 43 32 37 32 33 00 00 00 00 c0 7b 6d 38 36 27 00
 0000c0 00 00 00 05 00 51 89 41 00 66 12 6d ff 00 00 00
-0000d0 00 06 00 00 00 47 43 32 42 37 31 00 00 00 00 d0
-0000e0 89 6d 38 36 27 00 00 00 00 06 00 04 96 41 00 12
+0000d0 00 06 00 00 00 47 43 32 42 37 31 00 00 00 00 c0
+0000e0 7b 6d 38 36 27 00 00 00 00 06 00 04 96 41 00 12
 0000f0 0a 6d ff 00 00 00 00 06 00 00 00 47 43 33 30 39
-000100 46 00 00 00 00 d0 89 6d 38 36 27 00 00 00 00 07
+000100 46 00 00 00 00 c0 7b 6d 38 36 27 00 00 00 00 07
 000110 00 ca 85 41 00 6b e6 6c ff 00 00 00 00 06 00 00
-000120 00 47 43 33 31 37 41 00 00 00 00 d0 89 6d 38 36
+000120 00 47 43 33 31 37 41 00 00 00 00 c0 7b 6d 38 36
 000130 27 00 00 00 00 08 00 5a 93 41 00 21 f1 6c ff 00
 000140 00 00 00 06 00 00 00 47 43 33 31 37 44 00 00 00
-000150 00 d0 89 6d 38 36 27 00 00 00 00 00 00 00 00 00
+000150 00 c0 7b 6d 38 36 27 00 00 00 00 00 00 00 00 00
 000160 00
 000161
+ echo ERROR binary comparing /tmp/gpsbabel.7545/lowrance1.usr reference/lowrance.usr
ERROR binary comparing /tmp/gpsbabel.7545/lowrance1.usr reference/lowrance.usr
+ exit 1
+ rm -fr /tmp/gpsbabel.7545


Thanx!
RJL


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
Gpsbabel-code mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gpsbabel-code
Reply | Threaded
Open this post in threaded view
|

Re: Garmin GPS database reader

Robert Lipe
In reply to this post by Olaf Klein
Olaf Klein wrote:

> this is my first try to support newer versions of mapsource, now
> called Garmin GPS database. Because this new database version is a

Thanx for tackling this, Olaf.  You're a brave one.

The current mps code has made me reluctant to tackle high profile,
reverse engineered, binary formats that change rapidly.  While we
haven't had problem reports on this format in about about a year,
Mark Bradley (and, though surely to a lesser extent, myself) spent a
disproportionate amount of time working on .mps to make it usable.  I'm
not sure if we really finally got the water bailed out of it or if
everyone just abandoned .mps in favor of .gpx.

Ian Cowley did publish a document explaining his analysis of the .GDB
format at:

  http://groups.yahoo.com/group/Garmin_GPSmap_76C/files/Technical%20info/

but I'm not sure which of the multiple strains (yes, there have been
several .gdb mutants so far. no, I don't know how much of the worlds
data is stored in any one of them) it actually correlates to.  It's been
a while since I read it, but I recall being impressed with the detail
and clarity in the doc.  It was very thorough.


Anyway, these are the reasons that made this format the #3 question on

        http://www.gpsbabel.org/FAQ.html

> Most problems i had was the sequence of route points. I've found no  

Routes were a recurring source of problems in the existing mps code,
too.  We found wild variations that I don't recall us ever really being
able to explain.  


I think you were wise to disentangle the code from mapsource.c.    

gdb_find_desc_from_icon_number is exactly
mps_find_desc_from_icon_number, right?  Feel free to use that function
directly - four other Garmin-centric formats do.

I tweaked the README slightly (references to "newer" don't stand the
test of time well) and made some minor edits before committing it.  

Please do add some kind of test case for it just so we can be sure
we have all the endianness (those Apple and Sparc users...) and code
coverage (leaks, bound violations, etc.) cases exercised in an automated
way.

Thanx,
RJL


--
Support GPSBabel by helping to improve it or fund those that that have
done so.  Visit:

        http://sourceforge.net/donate/index.php?group_id=58972


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
Gpsbabel-code mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gpsbabel-code
Reply | Threaded
Open this post in threaded view
|

Re: Lowrance Patch

rnlnero
In reply to this post by Robert Lipe
Robert Lipe wrote:

> I've committed this, but it does fail for me.  (No, I'm not happy about

Strange.  Are you using the new *.usr files I sent along with the patch?
They're binary files and need to replace the ones in reference before testo
will pass.

Ling
--




-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
Gpsbabel-code mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gpsbabel-code
Reply | Threaded
Open this post in threaded view
|

Re: Lowrance Patch

Robert Lipe
> > I've committed this, but it does fail for me.  (No, I'm not happy about
>
> Strange.  Are you using the new *.usr files I sent along with the patch?

Yes.   Please update and confirm.

> They're binary files and need to replace the ones in reference before testo
> will pass.

I missed the commit on all enchilada and ignoreicons the first time
around, but the failure shows up in the one that was updated and
committed.

$ ./gpsbabel -D2 -i lowranceusr -f */lowrance.usr
GPSBabel Version: 1.2.6_beta06162005b-isaac
LOWRANCE data_read: Major Version 2 Minor Version 0
LOWRANCE data_read: Num waypoints 9
LOWRANCE data_read: object_num = 0
LOWRANCE parse_waypt: Waypt name = GCEBB Lat = 35.972030 Lon = -87.134695 alt = 0.000000
LOWRANCE parse_waypt: creation_time (GMT) 12/31/129 9:0:0

LOWRANCE parse_waypt: creation time (local) Mon Dec 31 03:00:00 2029

LOWRANCE parse_waypt: base time (local) Fri Dec 31 23:00:00 1999

LOWRANCE parse_waypt: waypt time (local) Fri Dec 31 22:00:00 1999


These times seem pretty odd.   If the failing offset happens to be `time',
that would explain it.   Actually, I think it does.   Look at our failed
case:

$ ./testo
--- /tmp/gpsbabel.9775/bc1      2005-06-27 16:19:46.000000000 -0500
+++ /tmp/gpsbabel.9775/bc2      2005-06-27 16:19:46.000000000 -0500
@@ -1,24 +1,24 @@
 000000 02 00 00 00 09 00 00 00 ff 57 41 00 3c 7d 6c ff
 000010 00 00 00 00 05 00 00 00 47 43 45 42 42 00 00 00
-000020 00 d0 89 6d 38 36 27 00 00 00 00 01 00 95 97 41
+000020 00 c0 7b 6d 38 36 27 00 00 00 00 01 00 95 97 41
 000030 00 7d 42 6d ff 00 00 00 00 06 00 00 00 47 43 31


Breakpoint 3, lowranceusr_parse_waypt (wpt_tmp=0x99d4368) at lowranceusr.c:311
311             waypt_time = le_read32(&buff[0]);
(gdb) n
312             wpt_tmp->creation_time = base_time_secs + waypt_time;
(gdb)

Breakpoint 2, lowranceusr_parse_waypt (wpt_tmp=0x99d4368) at lowranceusr.c:314
314         if (global_opts.debug_level >= 2)
(gdb) x /4b buff
0xbfed236b:     0xc0    0x7b    0x6d    0x38

Flip the endinness around and we see the differing fields are indeed the
timestamp.  Further, if we subtract them, we're off by exactly one hour
between what we read and what we wrote:

(gdb) p 0x386d7bc0 - 0x386d89d0
$4 = -3600

It looks like you have a timezone leak somewhere:

 TZ=CST ./testo | head -9
--- /tmp/gpsbabel.10636/bc1     2005-06-27 21:33:42.000000000 +0000
+++ /tmp/gpsbabel.10636/bc2     2005-06-27 21:33:42.000000000 +0000
@@ -1,24 +1,24 @@
 000000 02 00 00 00 09 00 00 00 ff 57 41 00 3c 7d 6c ff
 000010 00 00 00 00 05 00 00 00 47 43 45 42 42 00 00 00
-000020 00 80 43 6d 38 36 27 00 00 00 00 01 00 95 97 41
+000020 00 c0 7b 6d 38 36 27 00 00 00 00 01 00 95 97 41
 000030 00 7d 42 6d ff 00 00 00 00 06 00 00 00 47 43 31
-000040 41 33 37 00 00 00 00 80 43 6d 38 36 27 00 00 00
(robertl) rjloud:/home/robertl/src/gpsbabel.virgin
$ TZ=EST ./testo | head -9
--- /tmp/gpsbabel.10674/bc1     2005-06-27 16:33:46.000000000 -0500
+++ /tmp/gpsbabel.10674/bc2     2005-06-27 16:33:46.000000000 -0500
@@ -1,24 +1,24 @@
 000000 02 00 00 00 09 00 00 00 ff 57 41 00 3c 7d 6c ff
 000010 00 00 00 00 05 00 00 00 47 43 45 42 42 00 00 00
-000020 00 d0 89 6d 38 36 27 00 00 00 00 01 00 95 97 41
+000020 00 c0 7b 6d 38 36 27 00 00 00 00 01 00 95 97 41
 000030 00 7d 42 6d ff 00 00 00 00 06 00 00 00 47 43 31
-000040 41 33 37 00 00 00 00 d0 89 6d 38 36 27 00 00 00


Is that enough for you to go on?

--
Support GPSBabel by helping to improve it or fund those that that have
done so.  Visit:

        http://sourceforge.net/donate/index.php?group_id=58972


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
Gpsbabel-code mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gpsbabel-code
Reply | Threaded
Open this post in threaded view
|

Re: Garmin GPS database reader

Olaf Klein
In reply to this post by Robert Lipe
Hello Robert,

> Please do add some kind of test case for it just so we can be sure
> we have all the endianness (those Apple and Sparc users...) and code

as first step i've added a testo sequence and reference data into the patch
for the GDB reader.

O.K.



019a-gdb.patch (60K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Lowrance Patch

rnlnero
In reply to this post by Robert Lipe
Robert Lipe wrote:
>
> It looks like you have a timezone leak somewhere:
>
Sigh.  I was trying to be clever and adjust for the timezone on the reader but
it isn't going to work since I don't have tz info on the incoming USR data.

Here's a patch for only lowranceusr.c and the new reference usr files. (my "cvs
diff" is still showing the huge gpx reference files and I don't think you'd
want them again).

Ling
--


lowrance_only.patch (23K) Download Attachment
lowrance.usr (486 bytes) Download Attachment
enchilada.usr (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Lowrance Patch

Robert Lipe
> Sigh.  I was trying to be clever and adjust for the timezone on the
> reader but it isn't going to work since I don't have tz info on the
> incoming USR data.

It's not GMT?   Shame on them.

(For some reason, I feel like I'm in that movie "Groundhog Day" here
sometimes...)

> Here's a patch for only lowranceusr.c and the new reference usr files.

That returns the lowranceusr testo to a happy place.    I did some minor
cleanups to it and checked it in.

Thanx!
RJL


--
Support GPSBabel by helping to improve it or fund those that that have
done so.  Visit:

        http://sourceforge.net/donate/index.php?group_id=58972


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
Gpsbabel-code mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gpsbabel-code
Reply | Threaded
Open this post in threaded view
|

Re: Lowrance Patch

rnlnero
Robert Lipe wrote:


> That returns the lowranceusr testo to a happy place.

I'm not :-)  It occurred to me this morning I was on the right track with the
time calculations.  But I passed bad data to mktime(), plus some stupid logic
dealing with empty time.  So, here's a patch to fix that.

Also, the lowrance.usr in the repository seems to be corrupted?  It looks to be
truncated and it's different than what I sent.  Just in case, I've attached it
again.

Lastly, gdb.c is missing from the repository?  Or does it just need a day to
show up?  Without it, my updated working tree fails to Make.

Ling
--

lowrance.patch (2K) Download Attachment
lowrance.usr (486 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

README Patch

rnlnero
Lowrance blurb for README.

Ling
--

README.patch (685 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Lowrance Patch

Robert Lipe
In reply to this post by rnlnero
> > That returns the lowranceusr testo to a happy place.
>
> I'm not :-) It occurred to me this morning I was on the right track
> with the time calculations.  But I passed bad data to mktime(), plus

Happiness is a fleeting thing. :-)   I've committed this.  Thanx.

> Lastly, gdb.c is missing from the repository?  Or does it just need a
> day to show up?  Without it, my updated working tree fails to Make.

I plead incompetence.  Fixed.

It does take a bit of time (up to a couple of hours) between the time
I commit a change and the time it shows up in the public repository.
Yeah, it's pretty annoying during times like this when we have several
developers actively banging on things, but that's a sourceforge thing.


--
Support GPSBabel by helping to improve it or fund those that that have
done so.  Visit:

        http://sourceforge.net/donate/index.php?group_id=58972


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
Gpsbabel-code mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gpsbabel-code