Please please please learn about SQL Injection Attacks

Here are two more great blog entries about preventing SQL Injection Attacks

NOTE: This was rescued from the Wayback Machine. The original date was Tuesday, 30th November 2004.

Tags:

Original comments:

I think one of the problems is that there is too many source code archives and books that called their code “best practices” that are targets for SQL Injection. We will probably still see the code used for years to come.

11/30/2004 8:03 AM | Rocky Moore

Protecting Tables from SQL Injection Attack

A recent question in a forum that I view asked about how to ensure that even if one layer of security was compromised that the table would only ever return one row at a time so that an attacker would have to do more work to get a list of the users and passwords out of the database.

The way I see it, the best solution is not just to set up constraints, assuming that your database can add a constraint to only ever return the first row in a query, but to protect the table by not granting access to it directly. Then set up stored procedures to perform all the operations that you permit on the table. That way, if an attacker should get through the “outer defences” they cannot access the tables directly, and must use the stored procedures.

For example, say you have a database that has the user details for a website, this includes the user name and password. You don’t want an attacker to get a list of passwords or even one password. So you design the stored procedures so that you can pass a password in, but it will never put a password in a result set. The stored procedures for registering and authenticating a user for the website might be:

  • RegisterUser
  • VerifyCredentials
  • ChangePassword

RegisterUser takes the user name and password as parameters (possibly along with other information that would be useful for your website) and returns the UserID

VerifyCredentials would be used for logging into the site by accepting the user name and the password. If there was a match the UserID is returned, if not then a NULL value.

ChangePassword would take the UserID, the old password and the new password. If the userID and password match the password can be changed. A value that indicates success or failure is returned.

As you can see that the password is always contained in the database and is never exposed. The stored procedure could potentially generate a salted hash of the original password too so that should some layer of the database security be compromised that the password is still not readable.

You must also be careful when you call the stored procedure and ensure that you use parameterised queries. SQL Injection attacks are also possible when calling stored procedures if they are called by building up a SQL statement dynamically and executing it.

NOTE: This was rescued from the Google cache. The original date was Thursday, 6th January, 2005.

Tags:

Original comments:

Salted hash is a pretty tricky thing to do in SQL Server 2000, but should be very simple in SQL 2005 with a CLR function that wraps System.Security.Cryptography. For the moment you should probably hash the password on the client, i.e. in the web application code.

The downside, of course, to salted hashes is that you can’t ever tell the user what their password was. You have to have a facility to allow them to reset their password by supplying some other information instead (the ‘password reset question’ technique).

1/10/2005 12:04 AM | Mike Dimmick

Friendly Error Messages (or not)

Microsoft are normally quite good at producing friendly error messages when things don’t work out. However today I rebooted my machine after installing security updates, I fired up Visual Studio and then attempted to open the solution I was working on. Visual Studio then complained that IIS wasn’t running ASP.NET 1.1. So I went to IIS to check that it hadn’t reset my default website to ASP.NET 2.0, but it had. I changed it over to ASP.NET 1.1 and attempted to open my solution again. Same error message.

Curious I went back to the IIS admin tool and expanded the tree further to see if the Virtual Directory needed changing too. However, I then saw that IIS was stopped, so I attempted to restart it. Nope. Nada. It reported “Unexpected error 0x8ffe2740”. What the heck is error 0x8ffe2740?!

A quick Google found me a forum that discussed this and told me that it was because something else was listening on port 80, the default HTTP port. So, why didn’t the error message tell me this. Why the cryptic hex value?

Anyway, once I knew something else was using the port, I needed to find out what. I have a very useful piece of freeware called TCPView from Sysinternals and it is quite interesting to see all the processes with an open network connection. I quickly found the offending application (Skype, if you are curious) and closed it down.

NOTE: This was rescued from the Google Cahe. The original date was Monday, 17th January 2005.

Tags:

Original comments:

If you want to keep using Skype, go to File > Options > Connection tab and uncheck ‘Use port 80 as an alternative for incoming connections’.

1/18/2005 4:16 PM | Mike Dimmick

Tenets of Transparency

Eric Sink, Software Craftsman (and I really love that title) for SourceGear has written an article for MSDN about how ISVs can increase transparency and improve trust in their customers.

I especially like his comments on product licence enforcement as I used to work with a piece of enterprise software that had to be re-licenced each year. Usually a couple of weeks before the old licence would expire we’d get a fax with the new licence keys. The first time I came across this I was on my sandwich year at university so I got the great job of entering this information into server to re-licence the two-dozen or so components that were in use. Then I got to use one of the pool cars and go round to all the remote offices and repeat the process (with different keys – they were tied to IP addresses). This process took ages because often it would be difficult from the fax output to tell the difference between a 6 and a G, or 1 and I and so on.

A few years later I came in contact with this system again, but at least technology had moved on and the licence keys were sent by email. They still had to be entered by hand though.

Their next product release allowed a text file to be read in and the process was almost completely automated – Receive email, download attachment, open licence manager, open licence file, wait 5 minutes for it to go through the file and register the licences.

However it was still prone to problems. The licence manager code freaked out each October for one hour when the clocks changed. Some users of this system used it in emergency situations where an hour of downtime was unacceptable. Eventually the licence manager was changed to work on UTC times rather than local time. But a one hour downtime could have made the difference between life-and-death in certain situations. A power utility not being able to provide an emergency service to its affected customers, a water company not knowing quickly enough who is downstream of a contaminated supply and so on.

To read the article go to the MSDN website to read Tenets of Transparency

NOTE: This was rescued from the Google Cache. The original date was Saturday 5th February 2005.

Lean Software Development: An Agile Toolkit

Lean Software Development: An Agile Toolkit
by Mary Poppendieck and Tom Poppendieck
with Forewords by Jim Highsmith and Ken Schwaber

A review of sorts

The book defines a number of tools to assist you in implementing an agile approach to software development and uses a number of studies from both software development and manufacturing to get the points across.

On the whole the book is fairly easy to read and presents the ideas clearly with examples. However, in some areas I felt that the examples were too geared towards the manufacturing industry and not towards software development. This is, in itself, somewhat surprising as the authors have pointed out that you shouldn’t try and directly take a idea from one industry and apply it to software development without some element of thought. From the final page of the book comes a warranty that contains: “This warranty is invalid if practices are transferred directly from other discipline or domains without thinking, or if the principles of empower the team and build integrity in are ignored.” (p186)

The warranty quote above highlights two of the three areas that I feel are the most important. The other area was the “Contracts” tool. The remainder of this text are the things that I found most interesting, or provided one of those “a-ha!” moments as a realisation of how a new idea might make things work better occurred. Some of the things may seem obvious, but a gentle reminder of them is never bad.

Empowering the team means that everyone has a clearer idea of what is going on and, most importantly, why. It means that the team have access to the information to do their job rather than be hidden away from the client and given a point-by-point specification to work through. While it is easy to work through a point-by-point specification it does not allow the developer to highlight potential errors or conflicts if they don’t understand why the specification says what it does. There may be some missing information or an assumption may be wrong, or common sense doesn’t turn out to be as common as once thought.

Another point raised by the chapter on empowering the team was that “software development cannot be successful without disciplined, motivated people”. Motivation is not only the key to keeping good employees, but also for keeping them energised. The authors give the example of 3M who, for over 75 years, have allowed the “entrepreneurial spirit to flourish” by permitting “small, self-organising groups that become passionate about a possibility and are allowed to make it a reality… Scientists are expected to spend 15% of their time on projects of their own choosing” (p104). Many of 3M’s new product lines comes from this. In the software development world the equivalent is Google, who permit their developers to experiment with new ideas which Google then bring to the market place. This is very successful. A current example is Google Suggestion.

Going hand-in-hand with motivation is purpose. People need to have a purpose, a set of goals to work towards. This manifests itself in understanding why the software is needed. If this sense of purpose is to prevail then beware of the sceptics. “Nothing kills a purpose faster than someone who knows it can’t be done and has plenty of good reasons why” (p106)

Building integrity in ensures that the customer stays happy (perceived integrity) because the software appears to work as expected without errors or problems and it also ensures that the developers stay happy (conceptual integrity) that the code base stays well ordered and easy to maintain.

For a software application to have perceived integrity (p127-134) it needs to be usable, reliable, economical, and it must function as expected. In terms of developing an application in this way domain models are built, which are models of the software that the customer understands. The authors go on to say that eventually the models created will fall into disuse, and that they should not be maintained just because it “seems like a good idea”

In terms of conceptual integrity (p135-149), the “system’s central concepts work together as a smooth, cohesive whole.” Specifically “particular attention should be paid to the presentation layer, since conceptual integrity in the interface design is a primary driver of perceived integrity”. In order to maintain the conceptual integrity the system must be simple (many software patterns aim to bring simplicity to complex designs), clear (so that everyone in the software development team can understand), suitable (so that it fits its purpose), D.R.Y (Don’t Repeat Yourself – removing duplication means that if something has to be changed it only needs changed in one place), with no extra features (“Anticipating the future is usually futile and consumes resources”)

Finally there is the Contracts section (p161-177). For me, this was the key to the whole book. For many areas in the book I was left wondering how this would ever work in an organisation that exists to service clients. For many of the concepts I could see how they could gain a hold inside a software product company, but not a software services company. They gave examples of the traditional time-and-materials and fixed-price contracts and how they don’t really work well. Then they presented three examples of contracts that would fit into the Agile development process.

NOTE: This was rescued from the Googe Cache. The original date was Saturday, 11th December, 2004.

Tags:

What I've been up to

I’ve not posted anything for a while so I thought I’d show of some of my better photos that I have uploaded to Flickr (just to show you what I’ve been up to). These are my ten favourite photos that I’ve uploaded. These are in no particular order:

Cannon
This is a cannon on Calton Hill that points towards Edinburgh Castle.

There wasn’t a plaque or anything to explain it so I am assuming that it might have something to do with the one O’clock time signal. On Calton Hill there is the time ball that drops from the top of Neilson’s Monument so that ships in the Firth of Forth can see see it. At Edinburgh Castle a cannon is fired at one O’clock.

Edinburgh's Folly
Rear view of Edinburgh’s Folly, also known as “Edinburgh’s Disgrace”.

On the top of Calton hill is this is a 19th Century replica of the Parthaenon. Well, more accurately it is an unfinished replica – the money ran out before they could finish building it.

Industry at Grangemouth
Looking west over the Firth of Forth towards Grangemouth.

Helmsdale, Scotland
Helmsdale, Scotland.

Snow Dusted Edinburgh
Snow Dusted Edinburgh

Princes Street Gardens, Edinburgh Castle, National Gallery of Scotland

Edinburgh Castle
Edinburgh Castle at the start of Twilight

The Two Bridges
Looking west to the two bridges over the Firth of Forth

Sunrise Silhouette
Sunrise over Edinburgh’s Old Town

Forth Bridge (1)
Taken from the promenade at South Queensferry looking NE. From Wikipedia: The Forth Bridge is a railway bridge over the Firth of Forth in the east of Scotland, to the east of the Forth Road Bridge, and 14km (9 miles) west of Edinburgh.

The bridge is, even today, regarded as an engineering marvel. It is 2.5km (1.5 miles) in length, and the track is elevated 46m (approx. 150 feet) above high tide. It consists of two main spans of 1,710 feet, two side spans of 675 feet, fifteen approach spans of 168 feet and five of 25 feet. The main spans comprise two 680 feet cantilever arms supporting a central 350 feet span girder bridges. The three great four-tower cantilever structures are 340 feet (104m) tall, each 70 feet diameter foot resting on separate foundations.

Declaration of Independence
The quote is from the Declaration of Arbroath, commonly regarded as Scotland’s Declaration of Independence. Sir James Fergusson, formerly Keeper of the Records of Scotland, produced this translation to English.

The photo is of what is displayed on the wall at the Museum of Scotland: “As long as only one hundred of us remain alive we will never on any conditions be brought under English rule.” The original quote continues: “It is in truth not for glory, nor riches, nor honours that we are fighting, but for freedom — for that alone, which no honest man gives up but with life itself.“

NOTE: This post was rescued from the Google Cache. The original date was Sunday, 13th March, 2005

Tags:

Things I keep forgetting about FileInfo

This is going to sound like a real newbie post. But I keep forgetting this particular bit of information and I keep having to write little throw away applications to find out the answer. FileInfo has a number of properties and the MSDN description on them are almost next to useless.

Given the file C:\folder\file.ext

So there we have it. The things that I keep forgetting about FileInfo that MSDN just does not explain (except the Extension property, it at least does explain that one)

NOTE: This was rescued from the Google Cache. The original was dated Monday, 28th March 2005.

Normalising the data model

Sometimes I see on forums someone who is trying to get some SQL statement to wield data in a particular way but the data model is just thwarting their attempts, or if they do get something to work the SQL statement that does the job is horrendously complex. This tends to happen because the data is not normalised (or “normalized” if you are using the American spelling) to the third normal form. Denormalised data models tend to happen for two reasons. Firstly, because the modeller is inexperienced and does not realise the faux pas they have made in the model, and, secondly, because the modeller has found the a properly normalised data model just doesn’t have the performance needed to do the job required.

The Scenario

In the example scenario that I am going to present an private education company is looking to build a system that helps track their tutors and students. So as not to be overwhelming I am only going to concentrate on one aspect of the system – the tutor. A tutor may be multilingual and can teach in a variety of languages and they may also be able to teach a number of subjects. The Tutor table has joins to a table for languages and a table for the subjects. The model looks like this:

The denormalised data model
Partially denormalised data model

As you can see there are 3 joins from Tutors to Languages and 4 joins from Subjects to Tutors. This makes joins between these tables particularly complex. For example, to find out the languages that a tutor speaks then a query like this would have to be formed.

SELECT  l1.Name AS LanguageName1,
        l2.Name as LanguageName2,
        l3.Name as LanguageName3
FROM Tutors AS t
LEFT OUTER JOIN Languages AS l1 ON l1.LanguageID = t.Language1
LEFT OUTER JOIN Languages AS l2 ON l2.LanguageID = t.Language2
LEFT OUTER JOIN Languages AS l3 ON l3.LanguageID = t.Language3
WHERE t.TutorID = @TutorID

So, what happens if the tutor is fluent in more than three languages? Either the system cannot accept the fourth language it will have to be changed to accommodate it. If the latter option is chosen imagine the amount of work needed to make that change.

A similar situation occurs with the join to the Subjects table.

Solution

A better way to handle this sort of situation is with a many-to-many join. Many database systems cannot directly create a many-to-many join between two tables and must create an intermediate table. For those database systems that appear to be able to model a many-to-many join directly (GE-Smallworld comes to mind) what is actually happening is that an intermediate table is being created in the background that isn’t normally visible and the database takes care of this automatically.

The resulting data model will look like this

The normalised data model
Normalised data model

This allows a tutor to be able to register any number of languages or subjects. It also means that any joins on the data are easier as there are no duplicate joins for each Language or Subject. The above SELECT statement can be rewritten as:

SELECT  l.Name AS LanguageName
FROM Tutors AS t
INNER JOIN TutorLanguage as tl ON tl.TutorID = t.TutorID
INNER JOIN Languages as l ON tl.LanguageID = l.LanguageID
WHERE t.TutorID = @TutorID

This will result in one row being returned for each language rather than all the languages being returned into one row. It is possible to pivot the results back to one row, but currently in SQL Server 2000 that would add more complexity to the query than I am willing to discuss in this article. If you want to know how to pivot results in SQL Server 2000 then see the page on Cross-Tab Reports in the SQL Server books-online. SQL Server 2005 will allow PIVOTed results directly. For more information between the SQL Server 2000 and 2005 way of doing things see: Pivot (or Unpivot) Your Data – Windows IT Pro

Migrating existing data

Naturally, if you have existing data using the denormalised schema and you want to migrate it to the normalised schema then you will need to be careful about the order in which changes are made lest you lose your data.

  1. Create the intermediate table.
  2. Change any stored procedures using the denormalised schema to the normalised schema.
    • You may also need to change code outside the database. If you find yourself needing to do this then I strongly recommend that you read about the benefits of stored procedures.
  3. Perform an insert for each of the denormalised joins into the intermediate table
  4. Remove the old joins.

If possible the above should be scripted so that the database changes occur as quickly as possible as, depending on your situation, you may have to take your production system off-line while making the change. Testing the changes in a development environment first should ensure that the scripts are written well and don’t fall over when being run on the production database.

To move the denormalised Language joins to the normalised schema some SQL like this can be used.

INSERT INTO TutorLanguage
    SELECT TutorID, Language1 AS LanguageID
    FROM Tutors
    WHERE Language1 IS NOT NULL
UNION
    SELECT TutorID, Language2 AS LanguageID
    FROM Tutors
    WHERE Language2 IS NOT NULL
UNION
    SELECT TutorID, Language3 AS LanguageID
    FROM Tutors
    WHERE Language3 IS NOT NULL

It can, of course, be written as a series of individual INSERT INTO…SELECT statements rather that a large UNIONed SELECT

NOTE: This was rescued from the Google Cache. The original date was Sunday 3rd April 2005.

Tags:

Creating an informative workspace

Earlier this evening Agile Scotland ran a presentation by Rachel Davies called Creating an Informative Workspace which I found really interesting. The basic idea is that you permit developers to define their workspace to ensure that communication improves. The “Informative Workspace” is concentrated around “Big Visible Charts”. The following is a write up of my notes from the presentation.

The idea of the Big Visible Chart is to communicate the plan, the progress, the test results and the domain model. This is simply becuase it is too much information to be able to keep it all in our heads at once.

At the smaller end of the scale are the charts that each team member puts in their own space. These charts contain information that the individual developer finds useful, however they remain visible to all who want to view them. They become “Information Radiators”. That is, they radiate information so that other people can see the state of things without having to actually interrupt the developer. It is well known that an interruption that maybe only lasts a few seconds has a knock on effect as the developer then has to get back into the topic that they were concentrating on in the first place.

It is important that the developer is careful about what they present. The chart needs to be understandable so that people don’t go away with incorrect ideas or have to interrupt the developer. It also needs to be readable from a distance.

The team should be permitted to maintain their own space. Management should not bog down the developers’ charts with information that is not useful to the developers. If the information is not useful to the developers then it will distract them from the job at hand.

The team could, for example, use a team wall with lots of index cards pinned to it. Coloured stickers could be added to the cards that increase visibility and add information that can be easily spotted from a distance. At the more high-tech end of the scale a plasma screen could be used to display information such as the build results from cruise control.

The Big Visible Chart should also include the “big picture” so that everyone is reminded of what is being worked towards.  For instance, a scrum burn-down chart could be included; timelines to show what had been done and what is still to do; putting up the release date of the iteration and showing what is in the release and what is not.

All the information on the charts are a form of feedback that can be used to modify actions to keep everything on track. However, be aware of too much feedback which will overload the developers with information. If there is too much noise then people will filter it out and valuable information will not be recognised.

The process of creating the charts and defining what is put on them should be reviewed periodically throughout the project. It is important to have a “retrospective” meeting to discuss what went well, or not well, or is just puzzling. Update the charts as a result to ensure that the best infromation is available. These meeting could be about once per month, and the length could vary from an hour to an afternoon.

In the retrospective meeting do root cause analysis. Sometimes the process needs to be rethought rather than just speeded up as the observations made might just be a symptom of a deeper problem. Also, be aware of focusing too much on one problem because it is often possible that by moving another part on the solution to the problem becomes more apparent.

Getting positive feedback to developers is also important. If a developer can find out how useful they system they are building actually is to the users, or how it has benefited the business it can increase moral and improve productivity.

Some teams have mood indicators on their Big Visible Chart. This works well for indicating the mood of individual developers, however it isn’t so good at showing the mood of the whole team.

Since the majority of Big Visible Charts are paper based systems and have to be updated manually it is important to ensure that it is updated regularly. For example, there could be a daily standup meeting around the big visible chart where it is reviewed and updated. It is important that when information is no longer relevant it is removed.

NOTE: This was rescued from the Google Cache. The original date was Tuesday, 26th April 2005.

Tags:

Original comments:

Hmmm. I read this and I can’t but help thinking that it’s pop psychology coupled with poor information management. The worst thing about information, when working in a group, is that everyone expresses it differently. There’s a lot to be said about consistent approach, legibility, etc. That’s why the team lead should be the one orchestrating this stuff.

A lot of this is a rehash of the classic ghant chart showing milestones, who’s working on what, critical paths, who’s accomplished what, etc.

I guess the thing I’ve never understood about XP is that “they”, the XP proponents, feel they have to come up with all sorts of goofy communication tactics, like “scrum” and “mood indicators”.

XP is not a “one size fits all” solution. Nothing really is. This “Big Visible Chart” idea, sure, is a tool, in fact it’s something I’ve employed before XP was even invented. It’s not appropriate all the time though. Sometimes it can be a big time waster.

In my book, a better starting place would be to take the existing management tools and figure out how they need to be tweaked for the particular team makup, project requirements, and management style.

My 2c. Yeah, I guess I’m pretty opinionated.

4/26/2005 10:59 PM | Marc

Hmm… I guess i go more for Little Visible Tables. Gobs of ’em, tacked all over my walls. Get the colors and layout right, and i can pick out what i’m looking for just by turning my chair, no leaning necessary.

Oh, well, this doesn’t really do anyone else much good, but… by the time someone is standing near enough to my cube to read anything, they’re probably already asking me.

That said, having a great big whiteboard around can be a very handy thing on occasion.

4/27/2005 1:27 AM | Shog9

Marc,

XP is not Scrum. They are both different ideas within the Agile movement.

XP has never said it was a one size fits all. A common theme in the Agile movement is picking the correct methodology for the project. That is partly why it should be the team that chooses their workspace. Also, you said it should be the team lead that orchastrates “this stuff”. Remember that the team lead is part of the development team. It is important to get feedback from the team otherwise the developers in the team will feel that their ideas are not worth anything. If after some discussion some ideas get left at the side then that’s fine, at least the team thought about an idea and weighed it with the goals of the team. This is as much about moral boosting as it is getting information efficiently over to developers. If the information is no use to the developers then it is waste (another part of the Agile movement is Lean Development which seeks to eliminate as much waste as possible)

The problem with Gantt charts is that they become out of date very quickly. Especially as the project moves on and new information comes to light as the client is actually coaxed to ‘fess up about what they really want rather than the vague fuzzy requirements they came out with in the first place.

The “Big visible chart” is just a tool, I quite agree with that, and if I suggested otherwise then that was not my intention. It should be used along with many other tools.

Yes, Marc, you are opinionated 🙂 But that is good, because now I have to try and justify myself rather than just blindly accept what the presenter’s viewpoint.

4/27/2005 12:21 PM | Colin Angus Mackay

eXtreme Programming in .NET

This is a summary of a presentation by Dr. Neil Roodyn for the Scottish Developers that took place in Microsoft’s offices in Edinburgh on the 21st of July, 2005. At the end of the presentation I won a copy of Dr. Neil’s book eXtreme .NET: Introducing eXtreme Programming Techniques to .NET Developers which I started reading on the train to and from work today and my initial impressions are very positive.

What is eXtreme Programming?

XP is a set of five values, although only the first four are well known.

  • Communication
  • Simplicity
  • Feedback
  • Courage
  • Respect

Communication is very important, but it is often underrated. One of the important aspects is that everyone should be located together to improve communication flow. This is one of the reasons that Microsoft get all their developers together in Redmond.

Simplicity, or just keeping it simple, makes it easy to communicate and reduces the possibility of bugs. If the developers don’t understand how the code works then how are they going to understand what is causing a bug and, importantly, how to fix it without creating new bugs in the process.

Rigorous Feedback loops improves the software. Customers always ask for change but if they don’t see the software evolving the change request usually comes after much additional work has been done. This is prevalent in traditional software development. It is therefore important to show the customer the software frequently.

Courage is required as many of the XP practices seem hard and they don’t naturally make much sense. For example, making changes when they are needed or throwing away code. However, the XP values as a whole are like a safety harness that ensure that the project can proceed quickly and safely. It is important to ensure that the other values are adhered to as without them it would be like jumping out an aeroplane without a parachute.

Respect everyone in the team so that everything runs smoothly. The team means the software developers, the project manager, the customer, and so on – basically everyone who has an interest in developing the software. If there is no respect then poor software development results as the developers will grumble about the customer being stupid, and the customer will grumble that the developers don’t understand the business needs and so forth.  This lack of respect is also a symptom of a break down in communication.

Traditional engineering says that cost of change increases exponentially. This concept was stolen by software engineering but it is inappropriate. Consider, for example, the cost of having to move a concrete structure once it has set in comparison to creating the structure in the correct location in the first place versus the cost of moving a method in a piece of software. However, the traditional engineering approach has been pervasive in software development placing a burden on developers that simply does not exist if the XP values are taken on board.

Traditionally, software development has been “out of focus”. When a problem comes along the first thing that is thought about is the technology that can be used to solve the problem, however technology is just made up of features and toys. Then the process, that is the methodologies and best practices, are considered. Finally, the people involved are considered. However, this is the wrong way around. First and foremost the people should be considered first, then the processes, and then the technology. If the people working on the project are happy then they write better code and they tend to meet business objectives more readily.

What is software?

Software is just code in an executable  form. Code is just a set of instructions that tell a dumb box of silicon what to do. Code is the core of software development. The end result does not exist without code. However, it is often overlooked. Most companies write reams of documentation before any code is ever written.

In order to produce a better product it must be easy to install. The easier it is to install then the the easier the customer can test the product themselves.

The software must have features the customer wants. Often people are focused on unimportant things without realising it. XP has the “planning game” to ensure that the customer can create a priority list. This priority list can be changed at any point by the customer.

The software must be of high quality, which means that it repeatedly works. Every bug is treated as a high priority task so that it must be fixed before a lower priority task.

If every bug is treated as a high priority task then the need for a bug database is removed as new features are not permitted to be added until the bug is fixed which means that at any one point the known bug count will always be close to zero. Pair programming and frequent code reviews help find bugs early so they can be fixed early.

The attitude of the developers is very much different if they are adding yet another bug to a database with 500 bugs in it than adding a bug to a database with close to zero bugs in it. An analogy in the non-software world is the broken window syndrome. If you see a house that is clean and all the windows intact you may think is it a well maintained house. If one of the windows gets smashed and is not repaired quickly then more windows get smashed and perhaps some graffiti is added and the house will look dilapidated and rundown very quickly.

Also, to be a better product the software must also be upgradeable. This ensures that new features can be added easily.

It isn’t so hard to do!

If it isn’t so hard to do then why, depending on the statistics you read, do somewhere between 60% – 85% of software projects fail.

The main reason is likely to be politics. People may be are not interested in the software. They may a vested interest in ensuring the software does not get written, for instance, they may lose their job once the software goes live. There may be a lack of respect (see above) between the software developers and the customer.

Some companies base their business model on making money from the RFCs. They charge a lot more for the changes than for the initial development. They deliberately produce poor poor requirements and specification documents to ensure a high number of RFCs down the line. It is often the budgeting system in place in the customer that drives this area of poor quality.

A lot of energy is wasted arguing over petty things such as what language to use, what technology, or that the team should use X set of complex design patterns. It should be emphasised that a team should not use a set of design patterns just because they are there. Patterns should be used as a vocabulary to show what has been done, rather than what should be done.

A software company may impose a set of practices. But having one set of practices imposed over a whole company is counterproductive. The practices used should be tailored for each project. The practices must be examined to determine whether they will add value to a project or hinder it.

Why do developers make software that is so complex?

There are three main reasons for this. (1) is to make themselves look smart; (2) is to justify their “high” salary; (3) is to cover their backsides, for example, if they can exclaim “It was a tough project, look at how hard it was” then it can be used as an excuse if things fail.

Software Development the XP way

First and foremost do the simplest thing that could possibly work. Be careful not to interpret simplest as easiest. Simple does not mean easy.

Eliminate (or reduce) comments in the code. Comments are a sign that the code is unreadable and that the block of code being commented should be refactored into a method of its own with an appropriately descriptive method name.

Remove duplicate code. There are many patterns that can be used to remove code duplication. Once the code is refactored then it will be easier to read and there will be a single point in the code to change if the functionality is to be changed.

Limit the number of classes to only those that are necessary to get the software to work. It is not necessary to create unnecessary classes for “future requirements” as these may change and it would be extra work to alter these classes to fit the direction in which the software is going. Also, if old classes are no longer required then they must be removed.

As quickly as possible get feedback; interpret it; act on it. Feedback can come from many areas, for example the tests, the customer (via story cards or their response to a new iteration) or daily stand-up meetings.

Assume simplicity on day by day basis. Each day create a task list with the average task being about 30 minutes; some may take 5 minutes some may take 2 hours. If a task looks like it will take more than 4 hours then it needs to be broken down into smaller tasks. That way most problems will be easy to solve. If 90+% of tasks are easy then less than 10% will require more effort, but the ability to get through so many tasks and cross them off the task list will improve moral and improve the quality of the code.

Make changes incrementally because big changes don’t work as there is too much disruption caused and large changes are harder to understand.

Keep the quality of the work consistently high – the only two choices for quality level are “excellent” and “high”

Everyone should learn from everyone else. It is important to teach everyone to learn and think about how to teach others the information that you have. That way information flows around the team and enables everyone to contribute at a high standard all the time.

It is important to make sure that the software is in a state that is ready to ship on a regular basis.

Put in time for experimentation (called “spiking” in XP). Each spike should be limited to 4 hours. If it is longer than that then break down the initial task in to smaller tasks.

Everyone should be able to work in an environment where honesty and openness is encouraged. This aids communication and any problems can be avoided or fixed as quickly as possible.

Everyone should go with their instincts. They are there for a reason.

Everyone shares the responsibilities. For example, if a developer finds a bug they should fix it (or pair with the developer that created the code). They should not put it in a big database and wait for the other developer to come back from their holiday to fix it. Sharing responsibilities also means that everyone is “aligned” and going in the same direction.

Everyone needs to be adaptable because change is to be expected. Adaptability also means not carrying unnecessary baggage. For example, if the class is no longer needed then get rid of it, or if there is duplicate code then refactor out the duplication.

It is important to make realistic measurements of the time it will take to do something. Functionality is not complete until the customer is using it.

Back to basics

The basic stuff is:

Coding In this order
Testing
Listening
Designing

Without code there is no program.

Without tests then nothing is known about the quality of the program

Without listening the developers won’t know what the other developers are doing and won’t understand the business problem that is to be solved.

Without designing there is no organisation and no plane. But, designing is last on the list – Why do everything up front when it is going to change. Design just has to be for enough flexibility but no more. Too much design makes things more rigid and inflexible.

Iteration Zero

The very first iteration before any code is written is to set up the build machine to create automated builds and an installer for the software.

The Planning Game

This takes place during a customer meeting. The customer is asked to come up with a set of user stories. The developers then break the story down in to tasks that they can work on.

User stories are at the level of things such as “the user can log on to the system” which is a basic step the user would have to take to accomplish a larger overall task.

When the user stories are written down the customer must then priorities them. Preferably this would be stacking the cards in order, but if they are unwilling to commit to that level of detail then having the customer create, say, three stacks for high priority (critical and must be done to succeed), medium priority (software should have these implemented) and low priority (it would be nice to have these implemented).

Test Driven Development

Although Test Driven Development (TDD) is used by many, including Dr. Neil Roodyn, to mean “test first” this isn’t a universal accepted definition. Many people make a distinction between the two and use TDD to mean that there is testing involved and that development cannot proceed until the tests are written and pass.

Writing the tests up front means that the developer has to think about the interface more than the implementation. It ensures that the least possible solution is delivered. It means that the tests can be run the moment the code is written. It puts quality first. It helps the developer understand the problem better. And it gives the developer confidence that they are doing the right thing.

Refactoring

The purpose of refactoring is to allow the next piece of code to be written faster and provides a mindset of constant improvement. It ensures that code is reread and reviewed constantly which improves the quality. It makes life easier as refactored code is easier to read and understand. It also means that it is cheaper to add new features in the future as the code is clean and easy to understand.

Testing the GUI

It is possible to use reflection to drive the GUI in a test environment. There is also a very positive side effect that controls are named better from the start and that user feedback through the GUI is improved as the test framework needs to know what has happened.

Spiking the Unknown

When the developer finds an area that they don’t understand they need to explore it, experiment with it and be able to explain it to someone else (which is part of learning – see above)

Why do customers back off from XP?

Do they want the project to fail?

In fact, people still give Object Orientation lip service just as eXtreme Programming is paid lip service now. The reason is that these people don’t adhere to the values of the practice.

In some software development companies the business analysists  make great proxy customers. However, if the business analysists don’t really understand the business, and therefore don’t understand their job, then they get scared because they will be discovered as a fraud.

NOTE: This was rescued from the Google Cache. The original date was Friday, 22nd July, 2005.

Tags:

Original Comments:

This is why I think that while XP has some great ideas, they totally lose it on presentation. To quote from your blog (nice post, BTW, I’m not criticizing you at all):

Why do developers make software that is so complex?
There are three main reasons for this. (1) is to make themselves look smart; (2) is to justify their “high” salary; (3) is to cover their backsides, for example, if they can exclaim “It was a tough project, look at how hard it was” then it can be used as an excuse if things fail.

This is plain BS, IMO. Developers make software too complex because they’re taught that OOD is the cat’s meow, and programming “the Microsoft way” leads to a lot of awful design. It all comes down to education and experience, not puffing the feathers.

I wish XP would lose it’s own “attitude” and simply discuss the merits of its practices, and also talk about when and why and how XP fails. No system is perfect, and XP would be stronger if it looked at its own imperfections.

My 2c. 🙂

Marc

7/23/2005 12:37 AM | Marc

I accept your 2c and offer in return my tuppenny-worth.

I don’t take what you said as any form of critisism of me as this blog entry is, as I said at the top, a summary of a presentation that I attended. That said, I am a big fan of agile software methods (not just XP).

The presentation that I attended did discuss the weaknesses. In fact some of it is above: “A software company may impose a set of practices. But having one set of practices imposed over a whole company is counterproductive. The practices used should be tailored for each project. The practices must be examined to determine whether they will add value to a project or hinder it.”

So, if the practices of XP don’t fit the organisation then simply don’t use it.

As to other limitations – From what I’ve seen of XP, team member buy in is a big stumbling block. If they don’t buy in to the ideas then they’ll oppose it, even if they don’t mean to and are trying to stay neutral. The presenter gave an example of one team where one person just wouldn’t do pair programming – eventually he quit, even although the team were willing to accomodate him and permit him to code alone. However, if a more substantial number of team members don’t buy into the idea then XP simply won’t work.

Continuing with the example of pair programming. In the company that I work I have pair programmed on a small number of occasions. Each time the knowledge transfer was fantastic and we managed to get the work done in a fraction of the time. However, if I suggest it to most people they back off and say it isn’t necessary or it is a waste of time. I recon that a piece of work that took me almost a month to complete as I was stumbling through other peoples code would have been completed in less than a week if they guy who wrote the code, or was involved with it when it was created – however he wasn’t having any of it.

On to the “puffing the feathers”. I have to say that it is a problem that I’ve seen. In my younger days I may even have been guilty of it myself.

What is “programming the Microsoft way”? I’ve briefly looked as MSF (Microsoft Solution Framework) and it seems a reasonable way of constructing software. It is obviously biased towards MS technologies, but many of the principles can be transferred to other technologies. If, however, you are talking about some of these awful examples that can be seen in MSDN magazine on how to use some funky control then I agree that is often bad, but that is on a different level as it deals with the implementation and not the way the team works.

You mentioned that “XP would be stronger if it looked at its own imperfections”. I think that if you follow the values of XP then you are being critical of it all the time. The tight feedback loop provides a fast way of discovering what is wrong, not just of the software, but of the process and practices that are in place.

7/23/2005 1:02 AM | Colin Angus Mackay

I’ve often heard the problems of XP from the XP community, many of which are actually project problems rather than specific XP issues. To take a few….

1. Doesn’t scale to large teams
However, the majority of really large projects are probably a series of smaller projects if someone took the time to look hard enough.
The ideal XP team size is 12 or less. Can you imagine a billion pound health system being done by a team of 12! Perhaps XP and small teams is actually the answer to government projects?

2. Designing and coding for today
Although this is a strength it is also a weakness. Sometime we can take the time to code for something we’ll use tomorrow. This approach also means we can see significant refactoring effort as the project design evolves.

3. It requires good people
Any successful team requires a majority of good / experienced people to be part of the project. It doesn’t matter whether they are an XP team or not, a bad / inexperienced team are in danger of failing.

4. Distributed Teams
It is certainly more difficult to run distributed XP teams than using one of the more traditional development methods.

5. Feedback
Get effective communication between the business oriented client and the techies in the development team can be difficult.

I could go on and on. The weaknesses are actually challenges and many are present in non-Agile projects as well.

I have heard from many hardened XPers how and when it has failed them. Usually they learn the lessons from this and hopefully don’t repeat the same experience in future projects. All you need to do is ask the right questions of the XP community and they will tell you where it breaks or fails, sometimes suggestion alternatives or workarounds to mitigate the risks.

Remember, the original XP project was considered a failure!

Kent wouldn’t have brought out a 2nd edition of the white book if he didn’t want to modify and change some elements of XP to improve things and on occasions correct stuff that may be at risk of failing.

I think the previous commentator was referring to the flawed model of dropping components onto forms and running wizards, amongst other such intrinsically bad practices. This isn’t just a flaw of Microsoft – just about all the major compiler / tool developers has this poor software engineering approach to development. Oh! I’m in agreement with you. I hate the idea of a component that knows how to display itself, can control what it does and can do things with the persistence layer! This really is dreadful software engineering IMHO!

I agree that this is a really good post. Glad you enjoyed the events so much. It was rather good as someone else who was there.

Regards

John

7/24/2005 1:16 AM | John A Thomson

More information about this session can be found on Craig Murphy’s blog: http://craigmurphy.com/blog/?p=111

8/2/2005 7:16 PM | Colin Angus Mackay

I find this all very interesting.

I have been in the role of the client/business analyst on a complex software development project, using the XP programming philosophy for the last 8 months.

For people who own or manage a software development company here are a few observations about XP programming from “the other side”:

For XP programming to work, the development company needs to:

-commit to hiring programmers for their attitude and communications skills. They need to be able to talk to the client.

-look for programmers that are interested in learning, not just about technology, but about their clients business models. Programmers that are truly interested ask more questions, and produce results that better reflect what the client wants.

-find out what would make the client want to spend more time at their company (their own workstation, extension etc.)

-encourage interaction between the client and the developers in a non-work environment (arrange team lunches for the clients and the developers, without upper management)

It really comes down to building trust. This trust goes a long way with the client when things get rough (and a story estimated as 2 points turns into 8 points 😉

The programmers need to be gAs a client I believe I will get a better product from the developer who asks me 50 questions about my business, than from the developer who makes assumptions, even if the second developer is a more “skilled” programmer.

2. The development company should do what they can to encourage client, developer interaction. Here are a few things that have worked for our project:

– The development company that I work with has set up a fully equipped workstation for me in the project room. As a result I spend more time working there, where I am available to answer developers questions, than I do at my own company.

-The development team is given a budget for team lunches, and I am invited as a member of the team. These team lunches happen without upper management from the development company attending, which makes it a more relaxed environment.

-The development company arranges sporting events and invites the clients.

8/17/2005 7:25 AM | Jen

Thanks Jen, that sounds like some excellent advice.

8/17/2005 10:19 AM | Colin Angus Mackay