Pythonian Adventures 1

Wednesday, 30 January, 2008

OK, up front, I hate programming. I’ll agree that there is a great deal of satisfaction when a script or bit of software finally works and you can run a complex process automatically. And whilst I quite like the pseudo coding part (working out a solution to the problem), the actual process of getting the syntax, writing it and de-bugging it is incredibly frustrating.

That said, anyone “doing” GIS will at some point have to do some kind of coding and for many that will mean ArcGIS. ESRI, as with many companies that developed software in the 80s and 90s, have a history of scripting languages. And in scripting I mean taking existing software functions, making them accessible through a script interface and automating procedures. This was often developed to allow complex tasks to be “recorded” and re-run, but also allows looping procedures to be developed and therefore repeating a process iteratively (i.e. applying a buffer routine to 1000 files).

In ARC/INFO this was AML, in ArcView 3.x this was Avenue and in ArcGIS…. well they removed scripting. Of course by that point Microsoft was touting Visual Basic (and VB for Applications) and programmatically accessing COM Objects within software. This is a fantastically sophisticated approach because, with ArcGIS, incredibly low-level properties and methods are opened for access. Unfortunately it is not for the faint hearted and even the simplest of processes will require 50+ lines of code.

And the alternative? Well there WASN’T one. This was one of several “gotchas” that ESRI shot itself in the foot with when in released ArcGIS 8. Because whilst VBA allows for some really good extensions to be written (and I’ve blogged about Hawths Tools, Graphics and Shapes, X Tools ET Tools etc), the average GIS “analyst” may well not have the time or inclination to get that deeply involved. In these instances most people want to use a scripting language. O dear….

Where can you get a good coffee??

Friday, 25 January, 2008

A good question indeed and there seems to be a worry amongst coffee aficionados that the “big three” are suffocating independent (quality) coffee shops in the UK. In fact the BBC have a nice article entitled Giving it their best shot which is a nice write-up on “inferior coffee at an inflated price.” And the (partial) solution to this problem? Buy your coffee from a good independent. And where are they (OK, so there had to be a geospatial element)? Delocator provides a searchable database of coffee shops, overlain on Google Maps. The site is well designed, with a clean interface, through which you can also add new cafes. You can only search for full, or partial, postcodes which makes life a little harder when you are visiting somewhere, but other than that, “does what it says on the tin.”

NextMap Ireland currently in Edit

Thursday, 24 January, 2008

InterMap have been busy bees recently as the description of their current and impending imagery elevation products shows. For those who haven’t come across Intermap before, they are a Canadian company that specialise in flying airborne IfSAR and back in 2002 flew the whole of the UK for Norwich Union, acquiring imagery that was processed in to consistent 5m resolution DEM coverage of the UK. Well they have recently finished flying the entire island of Ireland (Northern Ireland and the Irish Republic), with processing currently taking place. And since the UK was flown, technology has moved on with 1m resolution now standard and I gather sub-1m data will be available for Ireland. It’ll be fantastic to see what this data looks like.

Landsat 5 back imaging again

Monday, 21 January, 2008

Back in September 2007 the USGS reported problems with Landsat 5’s onboard batteries which had caused it to stop imaging. Landsat 5 has proved really rather remarkable, proving very reliable since it started imaging in March 1984 (and completing its 125,000 orbit last year). It still provides Thematic Mapper imagery (at 30m resolution) and, almost single-handedly, has been NASA’s “Mission to Planet Earth.” Anyway, the USGS happily reports that the fault has been found and imaging is now back online. Testing is continuing to see how the satellite performs.

More on geometry

Saturday, 19 January, 2008

After my last post on the exciting new world of geometry that ArcGIS lets us access, I then stumbled across some new work from Jeff Jenness, of Jenness Enterprises. Jeff does GIS consultancy and has made available a nice selection of well written (genuinely useful!) Avenue scripts for Arcview 3.x. Well, the world turns and Jeff has been putting efforts in to learning ArcMaps complex VBA offering, producing a major extension called Graphics and Shapes.

The title belies what is a very extensive feature set, split across managing vector layers (points, lines and polygons) and managing graphic objects (selection, flipping, zooming etc. Jeff has included a “calculate geometry” function which people might think apes ESRI’s offering. Well, take a look because yet again ESRI are put to shame in offering access to basic functionality. And he does it with a very well designed series of screens and a progress bar that actually works! If you look at the features on offer, polygons, for example have area, centroid, perimeter, parts and vertices. Lines have length and azimuth. The latter is something that is very useful for all sorts of applications and, you’ve guessed it, ESRI don’t offer it as an option. Whilst I appreciated the momentum of ESRI products and vast array of scripts that are available, it is somewhat galling that you need to use an untested third party script to perform “first order” tasks. Particularly when ESRI charge such a premium. Anyway, getting back to the script, the real extra option here is calculating these values for a spheroid rather than directly from a projected dataset. When you start working over large areas, significant errors can be introduced using projected coordinates rather than working on a spheroid. See the Wikipedia entry on a Great Circle for a nice introduction. Great circles are used in aircraft navigation and are equivalent to “straight lines” on a sphere. Anyway, Jeff’s extension allows you to make these geometric calculations on a spheroid. Neat.

Of potholes and spatial literacy….

Thursday, 10 January, 2008

I hit a bad pothole in the village I live in a few weeks back, muttering the usual expletives about poor council services and then thought nothing of it. Being a model citizen, my wife said I should report it to the council. This got me thinking that maintenance reporting is a classic “web” activity and, not only that, is inherenetly geospatial: you need to know *where* to perform a rear. So off I went to Bedfordshire County Council’s website in the vain hope that I could achieve this relatively simple task. Much to my surprise, a “Pothole Reporting Form” was only about 3 menus deep and painfully simple to complete, emailing the request to the requisite department(indeed, they turn up and repaired the hole 3-days later. Local governbment at its best!). And….. no map. Yes, you have to tell them *where* the pothole is, but this is a written description which they work from. I can appreciate that a web form requires no specialist knowledge to set up, but there really should be a better interface for adding geospatial information.

This actually got me thinking about how people report information and, indeed, how it is requested. If you ask for anything for your own home you will usually give a postcode and house number. You won’t know that for anywhere other than your own home. If you phone the emergency services, particularly at the scene of an accident, a considerable amount of time will be use to get your precise location, which can also now be triangulated if you are using a mobile or, in the case of motorway phones, they know where you are. In these instances, people are specifically elliciting information about location to make absolutely sure. However location reporting is difficult if you don’t have a mobile or GPS, relying on the skills of the individual to geolcoation themselves. This is clearly a “spatial literacy” issue, something that is being addressed by the SPLINT CETL (Centre for Excellence in Teaching and Learning).

So, the “pothole reporting form” relies on the individual providing a verbal description of the location of the site, using some kind of linear referencing system (500m from junction with…. etc). Can they do this and what sort of errors on accuracy are there? Of course, we are dealing with a transport network, so there are a limited number of locations where the site can be, assuming there is no ambiguity about the town or street. The alternative is a map of some description. However this also requires a level of spatial literacy, arguably more so than with a written description (although I know which I would prefer to use). You would need to implement both a map and aerial photo interface (possibly also oblique) to cater for those less familiar with the area and even then they might not be able to locate the site. This would be an interesting exercise comparing the accuracy of geolocation reporting methods….

Global Mapper

Sunday, 6 January, 2008

Global Mapper is one of those little pieces of “GIS gems” that many people haven’t heard of and should have done. Global Mapper LLC was founded in 2001 and, from what I can tell, rapidly built a following of contented customers around the ability to import and export a vast array of vector and raster formats and visualise DEM data. Not only is it the Swiss Army Knife of file format converters, but it also has very efficient algorithms allowing fast visualisation and analysis of DEM data. And to cap it all off it is inexpensive (~$300) and has technical support direct from the developer that is honestly second-to-none.

So what does Global Mapper do? Well the partial list of supported formats currently stands at 92; if GM doesn’t support it then simply ask the develop. And remember it can import and export to these formats. Besides the usual useful list of candidates including all ESRI and ERDAS derivatives, MrSID, JPEG2000 and GeoTIFF, it also includes support for LiDAR (LAS) and KML, as well as live web downloads (e.g. TerraServer). Analytical facilities include live connection of a GPS, inter-visibility, 3D viewing, crop, reproject, merge, digitising, georeferencing, contour generation, triangulation, raster math… the list goes on.

What has really increased the profile of Global Mapper is a strategic alliance with Intermap that allows Intermap access to the source code and distribution to their NextMap terrain customers.

And the support? Well the best place to go is the Global Mapper Forum where the developer is very active. Expect responses to queries posted within the space of minutes to hours. And if it turns out its a bug or minor feature, expect a binary ready to download within 24 hours. For example, I had been loading some large LiDAR datasets (~5 million points or 100Mb) into GM, but it would fail to complete the triangulation of the 3D points. It turned out that the triangulation tried to complete entirely in memory and I didn’t have enough space. The developer changed the routine to allow GM to write to a temporary file on disk. Added to this is that forum members get 10% off GM or any updates.

So GM is an excellent toolkit and a GIS lab isn’t complete without at least one copy. It isn’t a full blown GIS and doesn’t pretend to be, but it is fast and have some fantastic capabilities. Note that it *only* uses triangulation to create TINs for the conversion of vector elevation data to raster formats. This may or may not be wanted (from an analytical perspective) and the rasterisation may well need the use of a product like Surfer, or R, round-tripping back to GM.