Government IT - central government and misunderstanding efficiencies of scale

Last week, the Commons Select Committee on the Use of IT interviewed Local and Central Government departments and the largest supplier to government which also was the only large supplier willing to take part as a ‘public’ witness. This week they questioned the government minister but I haven’t got round to watching that yet as the video stream requires me to watch in real time and be physically in front of a computer with the appropriate plugins installed. I would have preferred to be able to watch it at double time on the bus!

The local government representatives, though they may not be representative, largely demonstrated an understanding and engagement with issues even if, in one case, they found it hard to express that in ways that could be understood by the committee.

Central government departments

The representation from the central government departments gave me considerable cause for concern. If I were responsible for their current projects I would be asking lots of questions. Here are a number of the comments that rang warning bells:

1. 60% of the current project is being done in an agile way, the other 40% (core infrastructure) is not.
2. “legacy systems are not suitable for agile”
3. The advantage of using an existing supplier is that you can re-use skills and people and existing knowledge.

For clarity, I should explain why these rang warning bells:

1. One of the key benefits of agile is the ability to deliver a very early release of a product ‘end-to-end’ within a short time-scale. The act of doing this de-risks a project immensely and the act of failing to do this is a much less expensive way of discovering that something is wrong! You can’t deliver an end-to-end early release if the core infrastructure is not part of the system. So, there may be apparently valid reasons why this project cannot be 100% agile, but those reasons, or the decisions made in response are a sign of a problem.

2. It simply isn’t true that legacy systems aren’t suited to agile management / development techniques. In many respects, renewing (‘refactoring’) legacy systems is a task that benefits hugely from the principles of agile. In particular, agile provides skills and techniques for reducing the cost of change and one of the big problems with legacy systems is their cost of change. So, this quote, from one of the key leaders in the public sector, tells me that they simply do not understand what ‘agile’ means or can do.

3. Continuity is clearly a great way of retaining knowledge and skill. I think government should be doing a lot more to retain skill and I think procurement practices tend to enforce the throwing away of knowledge and skill, ultimately at the expense of the public who pay for everything. But you cannot, on the one hand claim to be running a procurement process that is even handed and at the same time argue that re-using the same supplier enables you to re-use skills, people and existing knowledge.

Integration, centralisation and efficiencies of scale

Anyway! All of these evidence sessions to date have thrown up enough material to write a book which is why it’s taken me another week to write anything coherent.

But, I’ve been saved because this week I reached page 57 of the “Written Evidence” document - the submission of Andrew Hardie which very effectively expresses my views on almost everything. To be honest, this is quite disconcerting at first, but also very encouraging as writing has never been my strong-point! To quote one of his points which I agree with wholeheartedly but which is a point made by few if any of the other contributers:

Perhaps, the greatest fallacy in both government and private sector ICT systems implementation is that greater systems integration is the answer. It isn’t. The more tightly you couple ICT (or, indeed, any) systems together, the greater the speed, range and impact of problems and side-effects become and the harder it is to change the resulting monolithic systems, precisely at the time when ever greater agility is needed.

There is a constant push to standardise systems in government to obtain ‘efficiencies of scale’ through more efficient procurement or through the wide use of identical software. This push for standardisation and efficiency almost always results in a decision to roll out a massive IT project. The conversation appears to go something like this:

A: We mustn’t have any more huge IT projects that cost too much and go wrong.

B: I totally agree.

A: Okay, so how are we going to reduce costs in IT projects.

B: Well, we are aware that there are lots of departments re-inventing the wheel with different IT systems that do fundamentally the same thing. We need to standardise the systems they are using and cut the outrageous costs of this duplication which is caused principally by a failure of communication between departments. The left hand doesn’t know what the right hand is doing!

A: Yes, I totally agree. It’s shocking!

B: Okay, so lets set up a project to consolidate all the systems together into one standard system.

A: Fine. Go ahead!

Yes, in the course of a few sentences they have moved from deciding not to have a huge IT project to setting up a new one, yet the flaw in the argument seems hard for government to spot!

There are all sorts of costs of complexity associated with interactions with people. Very rarely do multiple departments actually do the same thing. When you try to bring everything they do together into one system you typically end up with a system that is far too complex or a system that doesn’t meet the needs of the people you are producing it for.

The fundamental flaw in the argument above is the belief that different departments are actually doing precisely the same thing. In reality, if you talk to the users, you will find lots of differences in need and use and common practice. If you want to make the users more efficient at doing their work then you need the system to be tailored to their need not imposed on them from above.

The more complex flaw is that they criticise the departments for not communicating without recognising the additional cost and complexity introduced by requiring departments to communicate with each other! Failure to communicate is a cheap argument that is almost guaranteed to be accepted as grounds for criticism. But communication takes time and we almost universally suffer from too many meetings and not enough doing.

Government IT projects and managing risk

I’ve been trying to watch / listen to the witness sessions of the Commons Select Committee on the “Use of IT” that have been taking place over the last two weeks. The strongest theme that I detect in these sessions is the bemusement that past governments and past projects have failed to address the issues.

One word that keeps on coming up in the discussions is ‘agile’. As always there are many explanations of what it is and I can’t help but feel that none of the explanations seem to provide the select committee with the eureka moment that people need to understand what it means and why it is different from the status quo. I wonder whether an explanation in terms of the management of risk might help a little.

Agile project management and the management of risk

For the sake of understanding what agile means, particularly from the perspective of management and strategy, let’s consider IT projects and the management of risk.

There is a strong tradition in some areas of management that says that management is about control. In other words, a well managed project is a project that is well defined, fully costed and running to plan. If a project fails, then the response is that the project was not controlled enough; it was not properly defined at the beginning; it was not properly monitored throughout; it was poorly costed and lacked strong management.

This view of management has been the driving force for many standards and methodologies that government has adopted over the years to keep projects in check and ‘ensure value for money’. But as we all know, history shows that, in a lot of cases, this simply hasn’t worked.

In reality, so called government ‘IT projects’ are often massive ‘change projects with an IT element’. These projects are massively complex and inherently highly risky. Just like risk in the financial markets that risk can be managed, but however it is packaged up and hidden away it will not go away.

Let’s take this analogy of financial investment a little further. Imagine that you are given some money to invest in equity in an area of the market that is highly risky but has the prospect of significant returns. Do you:

1. Invest all you money in one company, locking yourself into an agreement that prevents you from selling your investment for five years; or

2. Spread your risk across a number of companies, ensuring that you can sell your investment at any time and re-invest it somewhere else if the market changes or you realise that you made the wrong decision in the first place.

Now, imagine that, for some bizarre reason, you take the first option and you invest everything in one company. Would you reduce your risk if you took two years to decide which company to invest in and you ask each company to set aside a considerable period of time and resource in making detailed projections of where the company would be in five years time?

Okay, I hope it’s clear to see that this analogy bears remarkable resemblance to the traditional government approach to contracting out IT projects. I also hope it is clear that, when seen as a task of managing risk, the second option is clearly the way to go.

The second option is, in effect, the ‘agile’ approach. You diversify your risk and you reduce your risk by simplifying a big problem into a large number of smaller problems. You push against those who say that a project is inherently large and find ways of building a small working part of it very quickly. You push against those who say that everything needs to be produced by one company or team and use existing open standards as a way of guaranteeing interoperability. In this way you prevent ‘lock-in’, you open up projects to a much larger number of smaller development teams, and you encourage cooperation, experimentation and innovation.

---

I hope, for a few people, this explanation might help to bring a better understanding of some of the problems of government IT projects and why an open and ‘iterative’ (under whatever name) approach is crucial if government is to manage its risks and deliver the real potent for a huge return.

Written evidence to the Commons Select Committee on the Use of IT

A number of recent articles have drawn my attention to the Commons Select Committee on the “Use of IT” which apparently started an inquiry today, but then again the web site said that yesterday too!

I’ve always had a very keen interest in the effective use of technology in the public sector, so I couldn’t help take a look at the subjects it was looking into and the written evidence it has received. Unfortunately, so far (and I’m only on page 15!), the evidence seems to reinforce the reasons why successive Governments seem to be so fundamentally unable to use IT effectively. The reason I say this, is that the evidence I have read so far demonstrates the massive diversity and contradictory nature of the advice they receive.

In this first blog on the subject I’ll touch on the suitability of management and delegation out of the public sector. I’m fairly sure there will be more posts to come on this who subject!

Suitable management

One theme that is already coming through in the first 5 submissions is the theme of control. Are the management suitably qualified to manage the projects:

Management of the Public bodies is at all levels recruited mostly from non-IT backgrounds. Typically the managers possess long experience of the needs of the Organisation, but not of the issues raised by the development of an IT system. This leads them to make errors of judgement on IT policy.

I think the background of the management is probably a bit of a red herring, but it is absolutely true that a manager of an IT project must posses the skills and knowledge to be able to manage the project and take control. A common characteristic of management of failing IT projects is the complete reliance they have on the guidance and work of staff and consultants below them. The best manager of an IT project is the manager who can, if she needs to, listen and talk knowledgeably and meaningfully with the ‘shop floor’ IT staff doing the work and with ‘shop floor’ public sector staff who trying to use it. This is not about a public relations exercise of having top management ‘come down and talk to the workers’ this is fundamentally about whether the manager in charge has the knowledge to understand the problem and talk the language of the people who are implementing it both on the side of the technology and on the side of operations and change management.

Delegation

Closely related to this issue of management is the issue of where you place responsibility.

There are many politically motivated reasons for deciding whether public projects should be carried out within government or in the private sector or even in the voluntary sector. However, ultimately whilst this can have a significant impact on the way in which the work is carried, it doesn’t address the fundamental issues of why IT projects succeed or fail.

This quote from one of the submissions is a perfect example of this misunderstanding:

Government already has a model for a highly successful, cost free, technology implementation that has reached 80% of the UK population. It is the National Lottery. The technology is complex, secure and costly. It cost the taxpayer nothing because government intelligently pulled the levers that shaped an opportunity the private sector would fund.

Here the success of this project is attributed to the government distancing itself both managerially and financially from the risks of implementing an IT project. But in reality this project possess very few of the fundamentally difficult characteristics which government normally has to deal with in IT projects:

1. This was nothing new. The UK was implementing a National Lottery on the heals of many other nations who had done this before. There are few areas of public sector IT which fit so neatly

2. This was a perfect example of the large scale delivery of a standardised product. The National Lottery was not an IT project to modernise hundreds if not thousands of previous lotteries scattered around the country and carried out in their own unique ways in circumstances which were also very different regionally. The National Lottery was a blank sheet of paper with the opportunity to implement the best, cheapest, most effective solution and to effectively impose it consistently across the country.

3. The requirements were relatively easy to define. Ongoing success was easy to measure. Ongoing success and incentives were aligned with ongoing government requirements.

In short, the current government may well have it’s own political view on who should carry out IT projects, but this decision should not be confused with the real issues of making projects work. If the government decides to pass responsibility to the private sector or even the voluntary (should we call it ‘open source’) sector, it will nevertheless remain responsible for the end result. Ultimately the Government must either take control or loose control.

"show your working"—a plea for Scala coding quality

The recent debate about the complexity of Scala over Java and other languages has lead me to bemoan the quality of some significant chunks of Scala code currently circulating in the public domain.

I probably should just be patient as the cycle of programming language maturity is a common one. In the early days of a language, in the absence of coding standards and common practice, people grab at the new features of a language and enthusiastically use them with little awareness of the consequences. As a language matures, the community of developers adopts and polices standards to the extent that a newby developer soon finds themselves instructed on the basics of what to do or not to do.

There's a lot of Scala code out there that feels like it's been put together by spotty adolescents with too many hormones and not enough common sense. Okay, I'm being harsh, I'm a Scala newby too, but I think it really is time we started to get a bit more focused on developing some good coding practice and mentoring each other towards maturity.

I'm going to start with a simple issue which is close to the recent complexity debate...

Extremes of pure functional style encourage code obscurity

Functional programming is great, but taken to an extreme functional programming can:

• encourage short meaningless function names.
• discourage temporary named variables which help to explain what is going on.
• encourage the creation of large numbers of small functions which obscure the modular interfaces.

Take this function as a slightly extreme example:

  def ^!!||^(implicit f: Foldable[IN]) :
      Kleisli[Option, String, NonEmptyList[List[Char]]] =
          kleisli((p : String) => {(this !!|| p).toNel })

I'm not trying to pick on anyone here, so apologies to the author, but this method is one of over 50 public methods in a single Scala trait of an open source Scala library. Is it any surprise that people think Scala is complicated?

So, here are my Scala programming tips for the day:

1. Minimise your modular interface: in other words, keep your publicly accessible functions, properties and classes, etc. to a minimum. If it doesn't need to be public make it private. Why? ... because developers reading your code shouldn't be left swimming around trying to find what's important and what isn't. If you create a function that is only used by one other function, define it within the function that requires it. This makes it absolutely clear that it is only there to support the other.
2. Use meaningful names and avoid symbols: if your readers can't figure out what a method is supposed to do then chances are neither will you in a few months time! The use of symbols for function names is a curse of the current trend in DSLs (Domain Specific Languages). Yes, do define a + method to add two objects together, but don't write functions called, ~>, ~~> and ~%> and think you're clever!
3. Don't be afraid to use vals to store partial results in your function rather than trying to string everything together into one long expression. Not just does this simplify the code by breaking it down into smaller parts, it also provides you with the opportunity to "show your working" by assigning partial results to values with meaningful names. I doubt an assignment to a val has any impact on performance. But I'll leave it someone more qualified in bytecode to back me up on that.

Okay. That's me done for today. Don't miss out on this great Scala community style guide. A really valuable contribution to Scala maturity.

The official way to bypass data modification on O2 mobile networks

Last week I received details from O2 of the official way of bypassing the 'optimisation platform' that O2 use on their mobile networks. They were particularly concerned that I pass on the following comment on the use of this and so I include it below verbatim:

when using this “bypass” function you must consider page impressions will take longer, on average, and thereby detract from the user experience, a slower experience. Also, a greater volume of data will be downloaded , on average, and those customers who do not have unlimited data will use their data bundle faster or incur high bills.

They then then went on to reference section 14.9.5 of the W3C HTTP 1.1 Protocol specification: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.5 which refers to the use of the HTTP Header "Cache-Control: no-transform" response directive. By setting up your server to return this response header, O2 indicate that they will not modify the data.

I should add that I haven't had a chance to test this yet and in particular, to test whether this stops the compression of images as well as the modification of HTTP source code.

Whilst this appears to offer a way for web developers to prevent their site content from being modified, it does not resolve issues for developers of web applications which utilise web data feeds that are not under their own control. For example, the developer of the great iPad Viewfinder app which provides Flickr photo search and download cannot prevent O2 reducing the quality of the images when using it over their mobile network.