A customer has an idea of what they want to get built. You meet, discuss. The ball-park figure is 6 months person effort and will cost $100,000. What happens next? Do you tell them you’ll see them in 6 months time with a product and an invoice?

I think it is one of the most critical factors to the success of a software development project that customers are on the same wavelength as the people they have hired to do the work. I’m talking about expectation management from both sides of the coin.

If you’re building an agile project (for argument’s sake let’s use 'scrum'), your customer needs to be around for sprint planning and review meetings. They need to be made aware of the time and frequency commitment required of their staff for testing. They need to know what can and can’t be updated in a Product Backlog. For that matter, has anyone explained to them what a product backlog is?

Having been involved in software development since the mid 1990′s, I’ve seen lots of changes in the way things are done. I’ve worked on projects where the 190 page business requirement spec was the bible, and programmers wouldn’t lift a finger until an equally impressive technical spec was completed. I’ve also seen projects where the client took up residence on the far side of the room and could be called upon at any time with fingers pointing to a screen and the question “So…is that what you mean?”.

I’ve worked with a number of agile techniques with different levels of success. Each technique has had it’s pros and cons from a programmer’s perspective or as a team leader I’ve occasionally had to go back to the manual and check if we’ve been doing things the right way. How long should a sprint run really be? Why do I feel like we’re pushing requirement changes uphill? Where’s the customer when you’re ready to deliver the latest batch of output? Oops, someone forgot to tell the customer.

If your customer is not fully aware of the needs you have of them and can’t provide the staff to fulfill those needs then your project is going to hit a wall sooner or later.

Tips for managing customer expectations are:

• Educate your customer in your way of working.
• Not all projects fit into a fixed methodology. There’s no point in trying to shoe-horn all projects into a methodology that won’t work.
• Ensure a customer based subject matter expert is available for requirements gathering and testing.
• Ensure mobile phones and other distractions are turned off during meetings.
• Draw up a schedule of the deliverables you expect to provide and the user acceptance testing time demands around these deliverables.
• If any blockers can be foreseen, tell the customer early. Don’t procrastinate the bad news.
• Try and develop a partnership with staff working as close together as possible.
• Stay open to changing requirements.
• Score changing requirements in terms of difficulty vs payoff. If it’s a 8 out of 10 in terms of difficulty to achieve but only gives a 2 out of ten in benefit, the numbers should speak for themselves.
• Try and keep your customer informed of what’s around the corner. If you have 2 week sprint runs, make sure they know what’s happening in a month’s time. This helps keeps minds focused on the big picture.

• Queensland Dept of Health payroll system that’s seen employees go underpaid, overpaid and without pay. Click here for more info
• LINK – The Victoria Police database switch-over that wasn’t. Click here for more info
• Myki – The attempt to introduce a single electronic travel ticket to Victorian public transport users.Click here for more info
• eHealth – An attempt to consolidate patient identification thereby enabling easier recording and tracking of patient events.Click here for more info

The details of these projects and their current status is available for anyone who wants to spend 2 minutes browsing with Professor Google, but suffice to say it’s not a pretty picture.

The upshot of each project’s failure is the negative image it portrays of the IT sector at large. Although the reasons behind the failures are different, the fact that each project is high profile and that profile goes even higher if it all goes to hell in a hand basket.

Diverse consulting companies, impatient politicians, slick tongued sales teams, uneducated buyers, an overly complex procurement and tendering system, a myriad of antiquated technologies. Take your pick, welcome to Australia!

It’s almost impossible to dismiss the fact that at the heart of all of the IT projects mentioned above is hardware and software.

As a software engineer, I’d like to think I build systems that stand the test of time. A customer buys code from me that fits their requirements, or can be configured to suit changing requirements. I don’t let my work out of the office until a test engineer has given it the stamp of approval and it’s got the necessary test cases to prove it. Occasionally you may find a difference between the expressed requirement and the executed version. That’s what we have user acceptance testing for.

You’ll find the same practices applied across most other engineering disciplines. It’s no surprise that some of the best programmers have come from the electrical and mechanical engineering schools.

So what’s going wrong? Why are so many public IT projects failing?

Are the engineers speaking up? Are we being listened to?

It’s interesting to note that the Australian government commissioned a Fellow of the Royal Academy of Engineering, Sir Peter Gershon to produce a report to tell them what they were doing wrong and how they could do things better.The Gershon Report has a number of recommendations for government ICT spending rationalisation and in many cases proper measurement of cost/benefits or IT projects.

I have no answers here, only questions.

• Have software engineers got a voice at project and program management level to make a difference?
• Have we got enough input at pre-sales meetings to create a realistic impression in our customer’s mind as to the reality of what they’re about to buy?
• Have we input into the skills that will be used to create or enhance the product or service being offered?
• Are we managing the expectations of key project staff correctly and is the message being passed up the line?

As a practitioner in the IT business, I don’t want to see any more debacles reflecting badly on my profession. But as I write the word ‘profession’ I am forced to ask myself have we reached the stage yet where we can honestly say we are professional, or are we just the same as the sales rep making their next buck?

I agreed to have a look at the project and was not forceful enough by allowing him to send an email to all and sundry welcoming me to the team.

I got access to the existing code-base over a weekend and spent a full weekend trying to get the site set up in my environment (a typical LAMP config). I threw my hands in the air on the Sunday evening and decided to wait till a programmer who had put in some quick fixes over the last few weeks came around to show me how to set up the website. I’ve been around the traps for some time now and setting up/administering linux, apache, php and mysql are second nature to me. Why should this site be any different?

Holy heavens above and all the saints preserve us if this wasn’t the site from hell. There were over 1900 files in the site (remember, the shopping cart wasn’t included yet..or sort of). There wasn’t a line of commentary apart from the standard headers the previous developers had copied and pasted from third party systems. In their wisdom, the developers had decided to use the .htaccess file not for it’s intended purpose (security and file access control) but as the core of their navigation system. In spite being built using a combination of PHP 4 and PHP 5, there wasn’t one occurrence of a 'Class' anywhere to be seen. When I turned on my E_ALL error constraint, I got more errors and warnings than actual web page when browsing the site, There was no revision control history so no idea of who’d done what or why?

I am usually very slow to criticise other programmer’s work. I realise I wasn’t there when the client asked for new requirements to be added on a Friday afternoon and expected them to work on the Monday morning. I wasn’t around when the lead programmer left for a bigger and better challenge, leaving the junior programmer holding the fort. That’s why I usually stay quiet when looking at other’s work.

This piece of work on the other hand really took the biscuit. The amount of code that went into producing the amount of output that was a handful of pages was staggering. I firmly believe the entire site could have been built (including a shopping cart) for less than $20K and should have taken 2 months to complete. The cowboys who built what I saw were screwing the site owner for every penny he had and measuring their delivery by the unnecessary lines of code they were producing.

In this instance I had to walk away from the project. I realised I would be the ‘goto’ man for all the sites many woes and with pending holidays would be unable to commit to deadlines as required by the client.

Unfortunately this sort of web development effort has left a bad taste in the mouth of the site owner as he now views all web/software developers with the same contempt and distrust. So thanks to the cowboys they have now driven an even bigger wedge between the world of software and it’s lifeblood – the customer.

As programmers, practicing our craft is a skill that’s fine-tuned over time, but the technology doesn’t stand still. It’s a programmer’s responsibility to ensure we keep up to date with new features, techniques and practices. Failure to do so results in mediocre code and a guarantee you’re not going to last the test of time as your skills age while the world moves ahead.

If you’re a software/web services buyer, please ensure you check the credentials of the people you are dealing with. Pick up the phone to other customers. It’s your money, you should be confident and comfortable handing over your cash knowing you’re getting a reliable and competent level of service.

If you’re a software or web developer, please be open to your clients. It doesn’t always work out well, but if you have a couple of successful projects under your belt, you should be able to use them (and the customer) as a reference. Expectation management is a key component of this business and it is especially true when you’re dealing with clients who are not tech savvy. It is your duty of care to educate, illuminate and leave a good IT experience in your client’s mind for all our sakes.

Oh, you’re a programmer. There’s your desk in the corner.

I’ll feed you three cups of coffee every hour and a pizza on Friday. Did you see the cool hip lava-lamp we have at our reception desk? You're really going to love working in our studioish office. Now, start coding! If you’re not sitting at your desk and we can’t hear you cutting code, we’re not going to pay you your assorted nuts at the end of the month.

Why not pay by the line of code

Maybe the scenario here is a bit pointed, but that’s the point I guess. There are three issues at play here.

Non IT management managing IT projects

I’ve seen it all over the place, the head accountant is appointed responsibility for the IT department (It’s a cost centre after all) and as such the decisions are made with this year’s budget in mind. That’s fine as long as you don’t have any projects that run into next year.

Management’s misconception that coding is all

If your programmers aren’t cutting code, what the hell are they doing? That’s the second time today I’ve seen him go for a coffee. This has got to stop!

Lack of understanding and then trust

If management has little or no understanding of the domain they have been given to 'manage', imagine the fear and mistrust in the conversation that takes place when the development manager tells them "We’re a few days behind because we had to redesign the hibernate files to take into account the changes in table structures brought about by the latest specification changes."

It’s might seem a paradox that in this, the age of I.T. off-shoring and the total farming out of the code cutting exercise, many organisations insist their staff sit in cubicles and make key-tapping noises to prove they’re working. But…it’s not a paradox. All that has happened is the task of the circus master has been delegated. The circus monkeys are still in their cubicles. The circus master is now taking his orders by phone or skype or webex.

In a perfect world

I’d roll out of bed and turn on my computer. After a bit of breakfast, I add a new component to the Trac project I’m using to manage my current freelance job.

The client has asked for a new feature to be added to the website I’m working on. She’s emailed me with a high level specification so I can estimate how much and how long it will take. I’ve provided the estimate and got sign-off. The email chain gets attached to the Trac component so the graphic designer gets an automatic notification of the new component’s go-ahead status.

I start working with a HB pencil and a bit of recycled paper on thoughts as to how the new piece of work will…work. After due consideration I sit down at the computer and start looking for some code that somebody else wrote and tested already that’s going to be close to what I want to do. The rest is easy.

It’s time for the 6 o’clock news. But not going to watch until I’ve update my on-line timesheet using Kimai with comments on what I was working on today. I see Joel the designer has updated his for 3hrs when he knocked off at 4pm after emailing me with the style sheet changes for the new component.

Component finished, ready for end user testing, uploaded to the test web server where the client can access it and report back (on-line through Trac) on changes and fixes needed.

Client gets an email every Friday showing her what was worked on, who worked on it and how many hours she owes us for.

The process is fairly simple, I have my working area in a ~/projects/myapp folder (where myapp is name of current project). I have lib files in ~/projects/java/lib/

I run the build in the myapp file, the ant file reads the build.properties file for some config info and dumps the build outputs to the $targetdir folder (~/projects/bin)

Firstly, the build.properties file

Now for the actual ant file. This is specific to a GWT application, but you can easily knock out the google bits and use the rest without any changes.

Feel free to suggest any ways to make this simpler and more automated.

I’d done the same from cvs to subversion in the early 2000′s and still think the move worked out well with a bit of preparation.

An unfortunate side effect of subversion is that it takes a good bit of effort to set up and manage a new repository so I ended up with two somewhat monolithic repositories…

1. archives
2. projects

The ‘archives ‘ repository contained all old project work and the ‘projects’ one contained relevant or current project work.

The result of this was that each archive had a bundle of sub-folders, each containing a project in its’ own right. Although git can handle the concept of sub-modules it’s not the best way to structure your project. In fact, it was a pretty lazy to structure my subversion projects in the first place but convenience overcame system administrative chores at the time.

This article explains the following:

1. Converting a monolithic subversion repository to a git repository
2. Breaking out the new git repository into a set of discrete repositories.
3. General backup process and scripts to copy the new repositories across to a backup system.

First off, a couple of conventions I keep are…

My server stores a central set of repositories with the name of myproject.git
My workstations work with their local version of repositories called myproject (no .git)
My server exports it’s .git repositories to a backup server using the same .git naming convention.
I have created a git.git user.group on my server to run all git processes. File ownership is given to this user so other users can’t fowl things up at least without thinking about it first.

Step 1 – Migrating to subversion

I store all my git repositories on a server in the directory

/usr/local/share/gitrepos

All of my subversion repositories are stored in

/usr/local/share/svn

Even though each folder is accessible using standard paths and commands, I have to go through the subversion door using the same mechanism I would access it from a remote box, namely http://. Other people may use svn:// if that’s their way of working.

Migration is taken care of with a single command.

An explanation of all the bits follow:

sudo – because I run everything as unprivileged user, sudo gives me the rights i need to create new files and folders.

git svn – this is the git subcommand that manages conversion of subversion repositories.

clone – make a copy of the repositories that I’m pointing at.

https://localhost/repository/projects – I access all my subversion repositories using secure http. This allows me to securely browse the content across the Internet and track things easily using Trac.

Depending on your setup, this parameter will contain the path to your svn repository,however you access it.

–stdlayout option tells git svn that my subversion repository is in the standard layout of trunk/branch/tags directories.

–authors file – this is a file I created by hand containing a list of all the people who committed project material in the past. It’d format is as follows:

-t tags -b branches -T trunk – these values are for completeness and I’m not sure they are necessary given the –stdlayout option, but if you’ve called your branches, tags and trunks anything different, this is how you find it.

Once the command has completed you should find a projects.git folder has been created and navigating into it you will see all your nasty subversion sub-folders which you should have set up as individual repositories in the first place…tut tut!

Step 2 – Breaking out Git into baby gits

The structure of the new git repository is something like this.

The .git folder (you cans see it using ls -a) contains a list of all the git related material such as project history, revisions and tags.
Check you can see your history by typing

The next task is to break the contents of the rather large ‘project’ folder into a git repository per project.
The tool for the job is a combination of the very useful git subtree command and a bit of custom shell scripting specific to this job.

The subtree functionality was written by Avery Pennarun and is hosted by the github site. You can download the git subtree command from the URL http://github.com/apenwarr/git-subtree. Installation instructions are on the same site so I won’t bore you with it here.

Once you have it installed you’re almost ready to roll. The following shell script has comments at the top to explain what it does. Save this to your favourite scripts folder and chmod it so it’s executable (chmod +x git_breakout.sh).

Then cd into the folder you created earlier (in my case /usr/local/share/gitrepository/projects). Run the script and watch the output. Depending on the size of your projects it will take a bit of time to complete; mine took about 20 minutes.

Once the script has completed, you should be able to see a new directory structure, something along the following lines.

Each folder is now its’ own git repository with it’s own internal version history, tags, branches etc.
cd into one of the folders and do a git log just to confirm you still have history.

The original projects.git can be removed (after backing it up for safety’s sake) if you’re satisfied everything is in place. We’re now ready to proceed with the next stage of the game, namely backing up our new git repositories to a remote share.

Step 3 – Backing up Git

I use a 2TB removable disk as a central data server for sharing files throughout my network. My thinking is that should the place ever burn down and I have the opportunity, there’s only one box I need to grab before rushing out the door. Of course, I burn frequent snapshots of this box to DVD.

I have created a ‘backups’ share on the disk which I map to my physical server using NFS.

The entry in my /etc/fstab file is something like this.

This means I can read/write to my backups folder as if it was a local folder on a local disk.

The job at hand is to be able to backup my git repositories using a cron task scheduled to run when everything is nice and quiet. Below is the script I use to run my git backups.

Once this script is run, you ‘ll have a copy of all your repositories in the backup folder. The next time it’s run should only take a fraction of the time as it won’t have to reestablish the git repository again.

So there you have it, migrating a subversion repository to git is really simple, the fun starts when you try to play with the results. I hope this has been helpful to others faced with the same job. Any improvements please let me know.

Imagine designing an information retrieval, indexing and presentation system for the two Mars rover vehicles that were sent on a one-way reconnaissance mission to Mars for a three month mission. The feisty little vehicles kept going for two years and the information systems had to be designed to cope with this unexpected project over-run.

The book runs to 168 pages and is a ‘should-read’ for anybody involved in designing or buying large-scale enterprise software. From an architect’s perspective, you get a reinforced mental checklist of the aspects of your designs that make them work and ensure they keep working long after you’ve moved on. From a customer’s perspective, you gain an appreciation of the effort put into designing such systems. From a developer’s perspective, now you know why you spend so much time writing and executing unit tests.

The book has short and well directed chapters and is an easy read with coverage of both the technical side of software development and the soft or human side.

As a result of this read, I went back to enhance some application logging classes that I’ve used on a number of projects to provide more granular output and statistics on usage patterns.

To develop the application you will need to download the Smart Device Framework libraries from OPenNETCF and add a reference to them in your project. I have included a feature that allows you to dynamically upload all necessary components to your mobile device. This is a cool feature enabled using a .Net custom installer application.

Feel free to download and modify the application to suit your fancy.

Fuel Monitor 1.1.0 Source Files

Fuel Monitor 1.1.0 Binaries

What this means is that every piece of data you accept from a user should be validated against a set of rules for that datatype.

Some general validation rules

If it’s a text string, does it contain any less than or greater than brackets?

If it’s a date, are it’s day, monthy, year parts valid numbers?

If it’s a credit card number, is there an online number validation algorithm you can use to verify it?

Regular Expressions in PHP

In order to validate your user-input, you’ll generally find yourself resorting to regular expressions. In PHP there are two options.ereg and preg. ereg is older, less sophisticated and somewhat slower than it’s younger brother. My preference is to use the preg options even though they’re a little more complicated, the end results is betterer(sic).

An example of using preg to validate a date:

An online manual is available from the main php website at http://www.php.net/manual/en/ref.pcre.php

And some lessons explicitly on how to use the preg function is available here http://www.php.net/manual/en/function.preg-match.php

Why does this happen in sites running on PHP? The answer is a duplication of escapes. Yep, a Houdini Supreme.

Firstly a systems administrator has installed PHP and set the value for magic_quotes_gpc = on in the system’s php.ini (usually located in /etc/). This will automatically add slashes to all GET/POST/COOKIE data. This makes it safe before writing it to a database. Mr O’Brein becomes Mr O\\’Brein when magic_quotes_gpc is set to on.

Secondly, a programmer has come along and thinking they’re doing the right thing takes all user input and uses the addslashes() funtion to escape all quotes. This results in a doubling of the escapes so, Mr O’\\Brien now becomes MR O\\\\’Brein.

When this data is rendered, we see the automatic removal of only one set of escapes but the other set is left behind…yuck!!

When programmers see this they think…”I’ll just use the stripslashes() method, I mean, that’s what it’s there for”. As the light from the idea bulb fades, they realise they’re fixing a problem that should never have occurred in the first place. You need to go to the source of your data and clean it up, make sure you’re either using magic_quotes_gpc=on OR addslashes. My preference is to use addslashes all the time and turn magic_quotes_gpc off, this way the logic of your code explicitly sets user input to be what you want.