igorbrejc.net

Alexander Ovsov has kindly translated my GroundTruth posts into Romanian. Quoting Alexander:

Our global volunteer project is called “Translation for Education”, located in all-around-the-world, with no headquarters. Its purpose is translating articles dealing with interesting science subjects such as biology, chemistry, geology, medicine, and information technologies in order to assist students and university staff who are not very good at foreign languages and help them become familiarized with relevant scientific news from abroad. Translation from English to small aborigine European (Indo-European) languages, not the big ones.

Thanks, Alexander, and keep up the good work!

Although I haven’t done any GroundTruth work for a long time, a couple of days ago I’ve started migrating (and upgrading) the source code from my local Subversion repository to Mercurial on Bitbucket. The code is in turmoil, but I hope I’ll be able to take some time from other work to clean it up and fix some bugs that have been found by users in the last year. I will also try to incorporate some new stuff that I’ve developed for Maperitive.

I’ve just released a new version of GroundTruth (dowload it from here) which contains fixes for bugs kindly reported by users in the last couple of months. I’m sorry you had to wait so long for this, but I’m dedicating most of my spare time working on the new Kosmos stuff. Anyway, here’s a list of changes for this release:

• BUGFIX: lat/lon were switched for generated OSM files…. ooops
• BUGFIX: the app. will now not fall apart if the user specifies an area which is not covered by (SRTM) DEM data
• the application now uses "neutral" (U.S.) culture for displaying console messages (no more decimal commas)
• Flee library is no longer used.

The new version of GroundTruth has just been released (download link). As I mentioned yesterday, the main work was on filling sea polygons based on the natural=coastline OSM tag. One additional feature is the ability to set the background color of the map (even if there are no coasts in your area). All of this is configured using the Options section of the rendering rules:

Sometimes it is not possible for the GT’s coastline processing algorithm to close up all of the coastlines and the resulting map can get ‘flooded’. In such a case I suggest you should use the new ‘–nosea’ option, which turns off sea polygon filling. Unfortunately a lot of the OSM planet extracts are lousy and they don’t include enough of the coastline to be able to connect it to the extract boundary, so I guess ‘-nosea’ option will have to be used quite frequently.

Some Background Information

This version of GroundTruth contains a lot of new (and improved) code which I was working on for the past several months. The work is primarily targeted towards a new generation of Kosmos tools, but since GT shares common geometry and OSM libraries with Kosmos, it can benefit from these libraries too.

Before going to bed I’m posting first screenshots of newly implemented coastline rendering in the next version of GroundTruth (due to be released tomorrow, if all goes well). The previous versions just rendered coastlines as lines, now the sea actually gets colored.

The maps shows parts of Isle of Wight, the screenshots were taken from Garmin Oregon 300.

NASA has moved the SRTM tiles again (they are now stored under v2.1 directory, but I don’t know if there’s any difference in the content compared to the v2 version). So GroundTruth’s and Kosmos’ contours functionality stopped working.

Unfortunately I’m in the middle of a major refactoring of the whole of my OSM code in preparation for Kosmos v3 and support for other sources of DEM, so it’s kind of difficult to produce running versions of my software: the builds are broken, I’ve temporarily stopped my build server etc.

But I’ve decided to take a few hours and prepare a new build for GroundTruth at least (Kosmos will have to wait, too much changes in the code). Since I didn’t have much time to test it, it is “officially” marked as an experimental build, and I’ve put it under downloads.igorbrejc.net/osm/groundtruth/experimental/ directory (just use the ZIP with the latest date). I hope it works.

BTW: if NASA decides to move the tiles again, there’s a new setting in the configuration file (GroundTruth.exe.config):

<add key="SrtmServerUrl" value="dds.cr.usgs.gov/srtm/version2_1/"/>

You’ll be able to change the URL yourself this time

For the past couple of weeks I’ve been busy on extending my digital elevation model (DEM) library with some new features, including the ability to use DEM sources other than the “standard” SRTM3. I’ve managed to produce a working version of the code which supports Viewfinder’s DEM1 data for Alps and I was dying to test it on Julian Alps, especially the area around Triglav, the highest mountain in Slovenia. The SRTM3 data for Julian Alps is very poor, with large chunks of height data missing. Here’s a sample contours map generated in Kosmos using that low-quality SRTM data:

As you can see above, the actual Triglav mountain does not have any contours. Compare that with the contours generated with Viewfinder’s data:

Not only all the contours are there (and seem to correspond with reality), but also the resolution is three times better than that of SRTM3. I’m pretty impressed with Viewfinder’s data, I have to admit.

This is only an intermediate experiment, I have to do some more work on the library before I integrate it into GroundTruth and Kosmos. But the results so far are very promising.

BTW: I contacted Jonathan de Ferranti, the author of Viewfinder’s site, and asked him about terms of use of these DEM tiles. Basically he gave a wholehearted approval of using it for open-source projects as long as the original authorship is respected. He does not claim copyright on these tiles since they originated from various sources, be he still likes to be informed about any large-scale use of them.

photo credit: State Library of New South Wales collection

I’ve succumbed to the Twitter craze (well, not really, just experimenting 😉 . You can follow me on twitter.com/breki74

Topics? Development, OpenStreetMap, Kosmos and other projects I’m involved in, hiking, photography, who knows what else…

I finally managed to integrate my new digital elevation model (DEM) processing code into Kosmos. It was a lot more work than I anticipated, mostly because the existing Kosmos’ source code is getting large, difficult to maintain and even more difficult to introduce new functionality. For me it is the best example of why SOLID principles are not just a developers’ buzzword, but that they do have an impact on maintainability of the code as the code base gets larger and larger. Although Kosmos’ code is highly modular, uses all the paradigms of object-oriented programming, even uses inversion of control (IoC) containers quite a lot, this is not enough: certain portions of the code are pretty bloated with multiple responsibilities, which is a first no-no when we talk about SOLID principles: keeping the design clean by separating responsibilities into separate services/components.

When changing the code which serves more than one responsibility, it is difficult to separate these responsibilities and difficult to test the impacts. Unit tests in fact become mini-integration tests and you typically need more setup code in order for a test to be able to run. Which produces unit tests that are fragile and, like the “main” code, difficult to maintain.

I mentioned the usage of IoC containers (Castle Windsor actually). The problem is that you need some mileage to get things right: a common mistake is to drag the container around in your code and use it explicitly to create objects. This defeats the purpose of the IoC container: to wire things together without your intervention. I agree with Joshua Flanagan when he says that the “container” word was unfortunate choice: it should be called a “composer”. Kosmos was one of the first projects where I used an IoC container and it shows: there are a lot of things which I would have done differently now.

All the more reason for me to continue on my effort of rewriting the critical parts from scratch. But that’s a story for some other blog post.

Anyway, I hope the new IBF thing works. The bad news is that I needed to change/hack some code, so I cannot exclude a possibility of a bug or two. The other bad news is that the old “.dat” contours format is no longer supported, so you’ll need to recreate contours for your projects.

The good news, on the other hand, is that we switched to the NASA’s new SRTM server, since the old one has gone to retirement. And even more: the new IBF code is much better that the old Srtm2Osm one, the contours are now placed properly on the map, the contour generator now knows how to correctly ignore data voids and the contour files are much smaller.

I also made changes to GroundTruth to cut the produced IBF tiles exactly around the specified map boundary.

Oh yes, the download links:

• Kosmos: downloads.igorbrejc.net/osm/kosmos/ 
• GroundTruth: downloads.igorbrejc.net/osm/groundtruth/

Enjoy! And if you find the stuff I’m making useful, please consider a small donation. It will be used for books, software and equipment, all for the benefit of producing even better stuff in the future:

Donation Amount:
(Currency: EUR)

Thanks!

Good news: I managed to update my Breki.DemLibrary code to be able to download SRTM data from the new NASA server. The next step was to update the source code for GroundTruth in order for it to be able to use the new code.

After some initial testing, I think GroundTruth is ready for a new release, so here it is: downloads.igorbrejc.net/osm/groundtruth/

If you notice any bugs regarding the new SRTM code, please report back.

Kosmos is the next one to be updated, although I expect a little bit more work on it since it uses an older code base. Hopefully it will be done in the next couple of days.

The other good news is that while making these changes to the SRTM code, I decided to overhaul the whole library. The main reason make the support for other sources of height data like CGIAR-CSI, viewfinderpanoramas.com, SRTM1 etc. much easier and transparent to users. I already started implementing support for these sources, but it will take some more time before the thing gets finished.

The main idea is for the DemLibrary to automatically find the best source of height data for a given area. So, for example, if you’re interested in generating contours for Alps, it will know that the best height data can be found on viewfinerpanoramas (presumably) and will download these instead of the plain old NASA’s SRTM3 cells (which contain a lot of “holes” with no height data). The library should even be able to combine various different sources into a single DEM dataset.

NASA moved the server containing SRTM data to a new location. Taken from their README file:

The SRTM dataset has moved. You have apparently been accessing it directly, rather than through the link specified at JPL (www2.jpl.nasa.gov/srtm/). The new location is dds.cr.usgs.gov/srtm/.Note that the directory levels have changed, so that the top level now contains the version directories. Lastly, we now only allow HTTP connections, not FTP. Sorry that we had no way to communicate these changes other than cutting off access at the old location.

The fact that they no longer provide FTP access is a big PITA for me, since all of my software (Kosmos, GroundTruth) uses FTP protocol to download the height data. I’ll need a few days of free time to write the new HTTP code, so unfortunately until then – I suggest you keep your SRTM caches on your disks.

As for Srtm2Osm, I’ll try to integrate this newly written code to it, but I cannot promise it will work – the code base has changed a lot since the time I published Srtm2Osm code on the OSM SVN server. Any volunteers?