Category Archives: Misc

Inserting geometry through a .NET Application

THIS POST REFERS TO THE NOVEMBER 2007 CTP (CTP 5) OF SQL SERVER 2008

Following from my previous posts (Getting started with Spatial Data in SQL Server 2008, Spatial Data in a .NET application) on the new spatial features of SQL Server 2008 I’ve been looking at how to get spatial data into SQL Server from a .NET application. This has not been as easy as expected.

I suspect I just have not found the right piece of documentation because my eventual solution isn’t one I’m entirely happy with.

I was unable to add a SqlGeometry as a parameter to the collection of parameters on the SqlCommand object in my .NET application. The SqlGeometry does not appear in the enumeration for SQL Server data types. My first thought was to put the data through as binary as I vaguely remember reading something about using binary in something I read recently, but I couldn’t quite remember where. However, when I created the geometry object in .NET then used .STAsBinary() on it so the parameter object would accept it the INSERT failed. The exception message was:

A .NET Framework error occurred during execution of user defined routine or aggregate 'geometry':
System.FormatException: One of the identified items was in an invalid format.
System.FormatException:
   at Microsoft.SqlServer.Types.GeometryData.Read(BinaryReader r)
   at Microsoft.SqlServer.Types.SqlGeometry.Read(BinaryReader r)
   at SqlGeometry::.DeserializeValidate(IntPtr , Int32 )
.
The statement has been terminated.

 

Yes, that was the MESSAGE from the exception. The stack trace above comes from within SQL Server itself. There is a separate stack track for my application. (I’m guessing if you are familiar with CLR stored procedures you may have seen this sort of thing before. I’ve not used the CLR code in SQL Server before, so it is a new experience)

The solution I found was to mark the parameter as an NVarChar, skip the creation of an SqlGeometry object and use a string containing the WKT (Well Known Text) representation. For example:

cmd.Parameters.Add("@Point", SqlDbType.NVarChar) = "POINT (312500 791500)";

I mentioned earlier that I’m not all that happy with this solution. That is because if I already have an SqlGeometry object I don’t want to have to convert it to a human readable format and have SQL Server parse it back again. Human Readable formats tend not to be the most efficient way of representing data and the geometry type already has a more efficient (as I understand it) format. Although in this case I bypassed the creation of an SqlGeometry object in my code, I can foresee cases where I might have created an SqlGeometry object through various operations and would have produced a fairly complex object. In those cases I wouldn’t want the additional overhead formatting it into WKT, passing it over the network and parsing it on the other side.

If I find a better solution I will post it.

Improving performance with parallel code

While waiting for my car to get serviced today I finally managed to catch up on reading some articles in MSDN Magazine. One of them, on optimising managed code for multi-core machines, really caught my attention.

The article was about a new technology from Microsoft called Parallel Extensions to .NET Framework 3.5 (download) which is currently released as a Community Technology Preview (CTP) at the moment. The blurb on the Microsoft site says “Parallel Extensions to the .NET Framework is a managed programming model for data parallelism, task parallelism, and coordination on parallel hardware unified by a common work scheduler.”

What I was most excited about was the ease with which it becomes possible to make an algorithm take advantage of multiple cores without all that tedious mucking about with threadpools. From what I gather the extensions are able to optimise the parallelism across the available cores. A looping construct can be set up across many cores and each thread is internally given a queue of work to do. If a thread finishes before others it can take up the slack by taking some work from another thread’s work queue.

Obviously, if an algorithm never lent itself well to parallelism in the first place these extensions won’t help much. Also the developer is still going to have to deal with concurrent access to shared resources so it is not a panacea. Those caveats aside these extensions to the .NET will make the job of using multi-core machines to their best much easier.

Parsing OS Grid References

If you have ever been to an agile coding session you may have come across the concept of the coding kata. It is an exercise designed to improve coding skill by making you more aware of the different ways of building the same solution. It also tends to lend itself extremely well to TDD.

I was just looking at ways of converting OS National Grid References from their alphanumeric form to a purely numeric form and it occurred to me that it might make an excellent project for a coding kata.

So, what’s the deal with OS National Grid References. Well, they consist of two letters followed by a number of digits. For example NT2474. NT relates to a square 100km along each side. The first two digits represent eastings within that square, and the second two represent northings within the square. The complete reference gives you a square that is one kilometre along each side. Of course, you can modify this to produce larger or smaller squares. NT, NT27, NT245746. As The actual coordinate the grid reference resolves to is the south west corner of the square. Also, there are optional spaces between parts, so NT245746 could be written as NT 245 746.

There is a more detailed guide to the national grid on the Ordnance Survey website.

Spatial Data in a .NET application

THIS POST REFERS TO THE NOVEMBER 2007 CTP (CTP 5) OF SQL SERVER 2008

Following from my last post about Getting Started with Spatial Data in SQL Server 2008 in the database, how about actually using it in an application? Since that is where most data will end up at some point, it must be possible to use the spatial data in an application some how.

So, using the Town table created before and populated with our fictional village of Pitcardine I wrote this very simple program to see what actually came through ADO.NET to the end application.

static void Main(string[] args)
{
    SqlConnection connection =
        new SqlConnection("Server=(local);"+
            "database=SpatialSandbox;Trusted_Connection=Yes;");
    SqlCommand command =
        new SqlCommand("SELECT * FROM Town", connection);
    connection.Open();
    using (SqlDataReader reader = command.ExecuteReader())
    {
        while (reader.Read())
        {
            for (int i = 0; i < reader.FieldCount; i++)
            {
                object value = reader.GetValue(i);
                Console.WriteLine("{0} ({1}) = {2}",
                    reader.GetName(i), value.GetType(), value);
            }
        }
    }

    Console.ReadLine();
}

The result was:

TownId (System.Int32) = 1
Name (System.String) = Pitcardine
TownGeom (Microsoft.SqlServer.Types.SqlGeometry) = POLYGON ((0 0, 370 0, 370 160
, 200 250, 0 250, 0 0))
TownGeomText (System.String) = POLYGON ((0 0, 370 0, 370 160, 200 250, 0 250, 0
0))

At this point I’m looking around to find the assembly that I can reference in my .NET application. After some looking I found it in C:MSSQLMSSQL10.MSSQLSERVERMSSQLBinn however that directory is not guaranteed as it depends on your SQL Server installation. It would be nice if the assembly showed up in the .NET tab in the Add Reference dialog rather than require the developer to hunt it down. (I’m guessing that this is due to the CTP nature of the product I’m currently using)

Once the column value can be accessed as an SqlGeometry object in the code it becomes possible to query the object. However, the results are pretty much all SQL data types. For example:

SqlGeometry geometry = (SqlGeometry)reader.GetValue(2);
SqlDouble sqlArea = geometry.STArea();
double area = sqlArea.Value; // could also cast to double

Note that STArea() is a method, not a property as you might expect in this sort of scenario. I suspect that each time it is called the area is worked out which may be an expensive operation hence keeping the method syntax.

Getting started with Spatial Data in SQL Server 2008

THIS POST REFERS TO THE NOVEMBER 2007 CTP (CTP 5) OF SQL SERVER 2008

This post is probably going to be a wee bit random. After the running around over the last couple of weeks with the MSDN event (Sharepoint for Developers) in Edinburgh, trying to get the Developer Day Scotland website up, an invite to a VBUG event in Reading, the Community Leaders Day at the IoD, DDD6 and the TechNet event of Andrew Fryer‘s 8 Reasons to migrate to SQL Server 2008 I’ve finally got round to trying CTP5 of SQL Server 2008. I actually installed it in a virtual machine within hours of it becoming available for download, but it is only now I’m getting round to trying it out.

First off lets start with the way spatial data is held in SQL Server. There are two spatial types, geometry and geography. Although they sounds very close there is a fair difference between then and it is probably best not to confuse them. However, both types are very well named.

Geometry is a simple two-dimensional grid with X,Y coordinates. The British National Grid is an example of this. I would guess geometry would be most useful in systems where data comes in a specific planar/flat-earth projection, or where mapping of small areas (such as the internals of a building) are needed. Lengths and areas for geometry are easy to work out. The coordinates will have a unit of measurement attached, for instance the National Grid in the UK is in metres, so the distance between any two points can be worked out by simple Pythagorean maths and the result returned in the same unit of measurement as the coordinates.

Geography fits the spatial data on a sphere with lat/long coordinates. This is a better choice for international data or for countries where the land mass is simply too big to fit in one planar projection. However, it is important to realise that lat/long is still projected. There are various schemes for fitting a lat/long position to a place on the earth. It is important to know which is being used otherwise data from different sources may not match up. It is not so simple to calculate distances and areas on a geography type as the distance between two coordinates changes depending on where those coordinates are. For example, a line that is 5º from east to west is smaller the closer to the pole it gets with the largest distance at the equator.

According to the documentation geography also has some other limitations. No geography object may be greater in size that a hemisphere. When loading data into the database it will generate an ArgumentException if tried, and if the result of an spatial operation results in a geography greater than a single hemisphere then the result will be null.

Finally, before getting on with some code, a note on SRID (Spatial Reference Identifier). Each piece of spatial data must be tagged with an SRID. Geometry types can have a SRID of 0 (which means undefined) but geographies must have a defined SRID. By default geography types use an SRID of 4326 which equates to WGS84. Spatial operations can only occur between spatial types with the same SRID. The result of spatial operations between two pieces of data with different SRIDs is null.

With that brief introduction to geometry and geography how do you create data in the database.

CREATE TABLE Town
(
    TownId int NOT NULL,
    Name nvarchar(256),
    TownGeom geometry)

To populate the column there are a number of ways of getting the data in. Currently SQL Server supports WKT (Well Known Text), WKB (Well Known Binary) and GML (Geographic Markup Language). For other data types converters will need to be written. The following example shows WKT:

INSERT Town
VALUES(1, 'Pitcardine',
       geometry::STGeomFromText(
       'POLYGON ((0 0, 370 0, 370 160, 200 250, 0 250, 0 0))', 0));

It is also possible to use the more precise method STPolyFromText. Naturally the parser will be more strict about what WKT it accepts when using the more specialised methods. For example if the WKT for a line string is sent to the STPolyFromText method the error would look like this:

Msg 6522, Level 16, State 1, Line 2
A .NET Framework error occurred during execution of user defined routine or aggregate 'geometry':
System.FormatException: 24142: Expected POLYGON at position 1. The input has LINESTR.
System.FormatException:
   at Microsoft.SqlServer.Types.OpenGisWktReader.RecognizeToken(String token)
   at Microsoft.SqlServer.Types.OpenGisWktReader.ParsePolygonTaggedText()
   at Microsoft.SqlServer.Types.OpenGisWktReader.ReadPolygon()
   at Microsoft.SqlServer.Types.SqlGeometry.STPolyFromText(SqlChars polygonTaggedText, Int32 srid)
.

At present it doesn’t seem to be possible to return the column as GML as the method isn’t found. The documentation for the method doesn’t work either so I suspect it is a feature that isn’t ready yet.

GIS and Me

Years ago, when I left university I, along with some friends, set up a company to develop a GIS product. We then went after some venture capital and just around that time rumours filtered through that Microsoft were hiring some of the best minds in GIS and we thought “Oh, bugger!” our newly formed business would not be able to compete with Microsoft. We were left in a state of shock when Microsoft release MapPoint. It was rubbish! (At least it was rubbish as a competitor to our product.) You couldn’t do even half of what our product could do or our customers were asking for in terms of new features. MapPoint was strong on route planning, but that was the only competitive edge over our own product. We didn’t do route planning at all. MapPoint offered very little in the way of spatial processing and, as far as we could see, was just a very limited way to display data on a map.

However, it looks like things are changing. Spatial data support was finally released in the November CTP of SQL Server 2008. There is no graphical front end, but the processing ability is there and that is the more important part. It is easy to put a new front end on to the data. Microsoft have some solutions in that area already such as Virtual Earth and MapPoint. There are also other vendors out there who have their own, more powerful, front end solution.

My only reservation about these new abilities, at least in the UK, is that spatial data sets are very expensive. Unlike the USA the UK government holds the copyright on the data it produces and it then charges a king’s ransom for access to it. This stifles business’s ability to use spatial data effectively and it will have to resort to poorer quality datasets or being restricted in the ways that it can use the data.A business could, for example, use map data generated by OpenStreetMap. Doing that stifles the way the business can use the data as OpenStreetMap is released under a Creative Commons BY-SA (Attribution Share-Alike) license. This means that anything a company does with OpenStreetMap data must be attributed to OpenStreetMap (not an onerous condition, if they use OS data they must include a Crown Copyright notice) and the derivative works it creates must be covered under the same license – How many companies are going to be willing to do that?

According to the OSM website “CC-By-Sa do not force you to make any source data available. You are only restricted in the license you choose for distributed data”. On the face of it that sounds like companies can use the license so long as the map is kept internally. Well, I can’t really see how that is possible. If one employee creates a derivative work, say a sales chart, and passes it out at a meeting then it has been distributed and must therefore be under the CC-By-Sa license and once in that state the company cannot stop its distribution. Obviously I am not a lawyer, but that is my interpretation of the license.

Currently, I’m looking at ways to put together demos and articles on using the Spatial data types in SQL Server 2008, however I cannot us OS data because it is too expensive and I cannot use OpenStreetMap because I’m not willing to release my articles, presentations and demos (my derivative works) under that license. I’d also like to use UK data because that is where I’ll be doing my presentations/demos.

If anyone has links to vector data that I can use then I’d love to know about it.

Geek Dinners

A few days ago I was invited to a dinner by Tim Leung of VBUG in Reading. Thursday night I attended that dinner. It was a dinner with a difference.

First of all Tim gave a short presentation on some of the new things he’d picked up at TechEd. As it was a small group of us around a table at the back of the restaurant it was much easier to engage the speaker and the presentation was more interactive as a result. By the end of the presentation a number of conversations had been started based on what had been discussed previously.

Afterwards came the food. The conversation continued in its technology vein. Everyone had something to share and some advice to give.

What was most interesting about this over other geek dinners that I have attended is the combination of the presentation at the start and the small (less than 10) number of attendees. I think the combination made for an excellent more intimate evening and I think that everyone got something useful out of it.

This is something I’d like to replicate in Scotland and interestingly earlier this evening Frank Kerrigan, my co-conspirator at the west of Scotland branch of Scottish Developers, was suggesting the same kind of thing.

Watch this space – we will be arranging something in the new year.