Stuart Roebuck

What is the Public Sector Network (PSN): a cloud of confusion?

2011-10-18T00:57:00.001+01:00

Recent announcements on the ICT Infrastructure programme for the Scottish Government and press coverage have drawn my attention to the Public Sector Network programme. Read around the topic I have become increasingly concerned that this programme is seriously flawed.

In summary, the PSN programme is a programme for creating a single public sector network for all UK public sector organisations including Universities. It is a programme that appears to be driven by the need to save money and a recognition that the existing arrangements are often outdated and inefficient. However, the proposed solution is very complex and potentially costly despite being argued publicly as a mechanism for saving costs and introducing a level playing field for commercial suppliers.

But you can’t really explain this all in one paragraph, so here’s a more extended attempt to explain what this is all about, starting with a look at the background and the surprisingly ambiguous definition of what the programme really is.

Programme establishment

In early 2007, the CTO Council articulated a vision for a Public Sector Network described as a network of networks delivering the effect of a single network for the public sector. In July 2008, the Public Sector Network (PSN) programme was established by the UK Government, principally coordinated by the Cabinet Office. (PSN-OM p.10-11)

But what is it?

Below is a selection of definitions or visions of what the Public Sector Network is. These are mostly from programme documents and should give a flavour of how confusing this whole thing is.

A single, integrated infrastructure, delivered by multiple selected service providers

A ‘private network of networks’ for the public sector, addressing the various special; security, resilience, service and availability needs of public sector organisations

Global, including overseas posts and other international UK public bodies

A secure version of the Internet for the UK Public Sector

(PSN PowerPoint Presentation)

The PSN vision is one of creating the effect of a single network across the public sector, to be delivered through multiple service providers in order to ensure ongoing value and innovation. In some respects, this is similar to the Internet model, whereby “service consumers” experience flexibility and inter-working without much concern for underlying inter-network “plumbing”. However, the vision is also one of a “private network of networks” for the public sector, addressing the various special security, resilience, service and availability needs of public sector organisations. (PSN-OM, p.10)

The Public Sector Network (PSN) will change the approach to the acquisition of Information and Communications Technology by the UK Public Sector, allowing public sector customers and select partners to harness changing technology to better support their delivery of service and the transformational government agenda. This will be achieved through a commonality of standards, a customer-centric operational model and a flexible approach. (PSN-PM, p. 1)

PSN is not a physical entity.
PSN is

an industry standard

an enabler for network interoperability benefits

an enabler to deliver procurement effort benefits

a process that provides a commonality

(SG-Breakouts, p. 7)

PSN offers a vision of ICT services from many suppliers being shared across the Public Sector and delivered over a common network infrastructure, itself provided by several network operators. (PSN-Comp, p.7)

The PSN is a supply-side “network of networks”, making network-oriented services utility-like for the public sector. Hence, it is essentially an inter-working and standards framework for the suppliers of network-oriented services to the public sector, governing both interconnection of supplier services and the relevant key service characteristics/attributes that ensure inter-working and end-to-end service assurance across supplier portfolios. (PSN-GCN, p. 7)

These definitions tend to raise more questions than they answer:

Is the PSN a separate physical network that duplicates the Internet but is intended just for the public sector?
When the PSN documentation talks about open standards does this mean that this is technically just using Internet standards or is it actually a network running under a protocol unique to the UK public sector?
Why can’t the public sector just use the Internet?
If this is separate from the Internet does this mean that staff in the public sector won’t be able to access the Internet?
What will this cost and how can it possibly save the government money?

Let’s try and tackle some of these.

So is the Public Sector Network a physical entity?

A breakout session of a joint Scottish Government, Socitm and Cabinet Office PSN workshop held in Edinburgh in January 2011 was clearly told that the Public Sector Network, “is not a physical entity”. But the extensive documentation around PSN includes description of the Government Conveyancing Network (GCN) - a part of PSN that “will be used to interconnect supplier data networks and other services in terms of network transport” (PSN-GCN, p. 8).

Similarly, the Operating Model document describes the PSN vision as “one of a ‘private network of networks’ for the public sector, addressing the various special security, resilience, service and availability needs of public sector organisations” (PSN-OM, p.10). If the PSN is not a physical entity then how can it possibly address issues of resilience and availability that are not ‘virtual’ concerns?

Looking at the various specification documents available on the Cabinet Office web site it is clear that the specifications include physical requirements like network timing and the implementation of domain name resolution across the network.

Ultimately the only way to make sense of all the documents and the conversations is to look at the terminology.

The truth, as far as I can tell, is that the PSN is the specification and not the network itself. Just like a car manual is not a car, the PSN is not a physical network but it does define the standards required to supply a physical network that is essential for it to run. To put this simply, whenever a document talks about the Public Sector Network you should probably insert the word “Specification” at the end. So, the phrase “The Public Sector Network is not a physical entity” should be read, “The Public Sector Network Specification is not a physical entity”, but parts of the actual network will be!

This is a very unfortunate and ambiguous choice of terminology.

What is the Public Sector Network Specification and what is the network it specifies?

So, the Public Sector Network (PSN) is the specification of a network that will deliver network services to users in a consistent manner throughout the country and even beyond the UK. These services will be physically delivered by private sector suppliers who have won contracts to deliver these services and who have met the compliance requirements set by government.

Third party private sector suppliers are providing something akin to an Internet connection, except it isn’t connecting directly to the Internet: it is connecting to a special public sector private network with what it terms, “segregated access” to the Internet (PSN-OM, p.21). This, the theory goes, allows the public sector to maintain a security separation from the Internet with communications running on networks which are provisioned to deliver higher resilience and availability standards than the Internet can provide.

So the idea is that the private sector will supply separate parts of this one single network and they will work together by virtue of adhering to the PSN specification. Hence, the PSN is not a physical network owned by the public sector: it is network that meets a specification and runs over networking hardware supplied to government by the private sector on a service contract.

Obvious!

So, why can’t the public sector use the Internet like everyone else?

This is a very good question. Indeed I would argue that this is ultimately the question that every government minister and civil servant should be asking repeatedly.

The proposed reasons why the public sector needs its own network appear to fall into the following headings:

For consistency across the public sector
For efficiency
For reliability and capacity
For security

Let’s take these one at a time.

Consistent networking across the public sector

The truth is that, to the present day, the public sector is a mish-mash of networks developed independently and often intended to be kept separate from each other. The need to bring these networks together to allow for sharing of information and intercommunication is hard to deny.

But given a choice does it makes sense to standardise on the networking standard of the world or create a unique one for the UK public sector?

Efficiency of delivery, procurement and maintenance

Maintaining the current range of networking services across government is undoubtedly more costly than it needs to be. It makes sense to standardise the platform so that procurement is less costly.

But what is the least costly option, servicing a network that is unique to the UK public sector or servicing a network like every other private sector business in the world?

Reliability and capacity

Enabling the public sector to share a common network that can balance the varying peaks and troughs of demand across services is a good way of dealing with capacity needs where individual and separate networks would be constrained by their individual capacities. Setting standards for the delivery of these networks can help to ensure that the networks are reliable.

But this concept of shared networking is the basis of the Internet. If capacity and reliability requirements genuinely demand more control there is no reason why the public sector couldn’t have a dedicated physical network running on the standards of the internet. In effect this is the model of the UK Joint Academic Network (JANET) used for many years by Universities and Colleges throughout the UK.

A secure network

Clearly the public sector deals with information of a private nature some of which concerns the security of the nation. So, do we need a separate network or unique networking standards in order to deliver the levels of security required?

It turns out that this question is answered by the specification of the PSN itself. It says quite clearly that it is possible to deliver all the security levels required over an “untrusted” network like the Internet.

Coupled with encryption technologies the authentication of individual devices will enable the sharing of information across the same PSN infrastructure from IL0 to IL4. (PSN-OM page 20)

(“IL0” (Impact Level 0) refers to “untrusted infrastructure” for example, the Internet. In other words, the PSN can operate over the Internet. So the special security needs of the public sector do not appear to mandate a PSN.)

The truth here is that much of the public sector has taken a “walled garden” approach to security. The idea is that you create a safe network within which public sector users can operate without worries of outside intrusion. Just like some secured government building, the security checks are made at the exits and entrances so that everyone inside feels safe to walk around with freedom and security. Unfortunately this isn’t really very secure at all. As soon as someone gets into the building they can access anything and everything.

So what do banks and security conscious companies do? They may provide a private network for employees but ultimately they make sure that data and systems that need to be secure are individually secured to the level commensurate with the data risks.

If the public sector took the same approach they could also operate on the standard internet like everyone else does.

So why are we doing this?

It is very hard to see any real justification for the Public Sector Network programme in its current form. So, how has it come into existence and why is it going ahead?

I can’t claim to know why the PSN programme happened but I can make some guesses:

In the absence of technical guidance it is quite possible that the public sector decision makers didn’t realise that they might be able to make use of existing technologies to replace all the complex networking systems that are currently in use. From that standpoint the idea of setting up a programme to design a whole new public sector network specification may have made complete sense.
Security always has a high profile and leads the public sector to demand the highest possible security standards in some things whilst completely failing to address the obvious issues like the unencrypted laptops, CDs and memory sticks that keep hitting the news. The security provisions in some parts of the public sector are so awkward that they practically force staff to ignore them in order to get their work done. Faced with pressures to be secure it is not surprising that non-technical decision makers probably ruled out the Internet as a viable option even despite then producing a specification that allows the Internet to be used!
The need to cut costs has resulted in a massive focus on centralised purchasing and procurement as an apparently obvious way of gaining ‘economies of scale’. This principal is much more widely applied than it is understood. Consequently it doesn’t surprise that government decision makers might feel that it would be better value for money to procure a government specific system for the whole of the country, than let individual regions procure an internationally standard system locally. In reality the latter is almost certainly more cost effective than the former and a lot easier to organise as well.

COSLA Police Summit - 23 Aug 2011

2011-08-23T22:29:00.000+01:00

Concerned by the way things appear to be going with plans for a single force in Scotland, I attended the COSLA organised “Police Summit” today in Edinburgh.

Here are some very brief notes:

It was reassuring to find that there were plenty of people there who were well informed about the limitations of the published reports presenting the various options. Colin Mair (Chief Executive, Improvement Service) did a particularly fine job of pointing out issues with the business case document.
I was shocked to discover that the leaders of Scotland’s main police bodies had seen nothing of the outline business case document. It put my failed attempt to see the document - as a member of the public - into perspective!
A running theme was that nobody knew what a proposed single force would mean in terms of accountability. Would it devolve power to local police or centralise it in the Scottish Police Board? Who would elect the members of the Police Board? How would the police interface at the local council level?
A number of people I spoke to one-to-one said that they recognised that we couldn’t stay with the eight forces model in the current economic climate. I asked why and discovered that they were just assuming that reducing the number of forces would reduce cost!
A number of the propositions and views seemed to suggest, more or less directly, that ACPOS (the Association of Chief Police Offices of Scotland) had historically acted as a barrier to change. There was a suggestion that, for some, the single force model had the attraction of bypassing the power of ACPOS. This provides a possible explanation why some may be pursuing a single forces model even if it doesn’t provide substantive cost savings over other options.
The consensus was that the SNP had decide to maintain its manifesto commitment to a single force and was unlikely to shift from this. However, there was also a suggestion that there might still be time to impact the government’s perspective on what a single force actually means.

A single Police force in Scotland, a single Fire and Rescue service in Scotland - a costly change for no good reason

2011-08-19T11:39:00.000+01:00

Why does centralisation save money?

A common theme in Scottish public sector thinking right now is the cost saving benefits of centralisation. If rumours are true then the Scottish Police forces, and the Fire and Rescue services are to be merged and centralised. Why? Because, it seems that there is a common unspoken understanding that centralisation, whether we like it or not, will save money.

You might imagine that reading recent government reports on these initiatives would explain some of the reasons why centralisation might bring savings. But you would be sadly mistaken. From the document "Sustainable Policing Project - Phase Two Report: Options for Reform" the nearest thing to an explanation of the benefits of centralisation is the sentence on page 5:

"The single force model represents the most significant change; however it provides the greatest opportunity to manage change, drive efficiency and in delivering operations when the change is complete. The eight force model represents the opposite."

In other words, there is no explanation of the benefits of centralisation. As readers, we are expected to accept - without question - that the author of this report is right in saying that single force is best.

This assumption is then reflected in the traffic light style summary of the report on page 62 (reproduced below). If you have a spare moment you might like to try (as I did) to go back through the document and uncover the justification for the colour coding of this:

Scottish Fire and Rescue

The Scottish Government consultation "A Consultation on the Future of the Fire and Rescue Service in Scotland" takes a similar approach to explaining the need for change when it introduces the subject on page 2:

"Many people, both in the SFRS and beyond, now accept that the current structure of eight Services supported by some national support services is unsustainable over the medium-term."

As readers we are clearly expected to agree with this statement because the people are unnamed and the argument is completely unsubstantiated.

Oddly, on page 5 of this report there is a nod towards decentralisation that is repeatedly scattered through the report from then on:

"To ensure that we can deliver the best possible outcomes for the people of Scotland, we want to see a context in which: the full benefits of the SFRS are de-centralised as far as possible…"

But the application of this principal within a single force is finally explained on page 21:

"A single service offers the best opportunity for de-centralising as many elements of the business as possible across Scotland. Under current structures resource cannot be transferred between services. A single service would give us the flexibility to de-centralise and move resource to meet specific, identified needs wherever they occur in Scotland."

In other words, the word "de-centralised" is included in the paragraph to make it appear that de-centralisation is happening when it's actually centralisation. Interestingly the paragraph also claims that current structure prevents the sharing of services when clearly this is not the case and the recent English riots is one well known example of this.

But back to the benefits of centralisation. On the same page this report says:

"A single service approach offers significant scope for bringing together funding streams and reducing the complexity of governance and the duplication of effort and control. It also allows for the rationalisation of back office and specialist services, but with the opportunity to spread back-office and specialist functions more widely across the country. Through simplifying the delivery landscape in this way, the costs associated with the existing multiple governance and co-ordinating mechanisms would disappear."

To be fair this looks like a reasonable attempt at a genuine explanation of the apparent potential benefits of centralisation. But the unfortunate truth is that the last 30 years worth of research in this area provides conflicting evidence of benefits of centralisation or decentralisation within the public sector. See for example: "Centralization, organizational strategy, and public service performance".

The idea that a single service will reduce the complexity of governance may be true, but it will not reduce the overall complexity of the problem. Centralised control allows for rapid decision making at the expense of a lack of detailed and localised knowledge. In effect it could be argued that any efficiencies of centralisation are in the way that it allows rapid and consistent decisions to be implemented nationwide regardless of whether or not they make sense at the local level. In practice centralisation is very likely to result in a nation of demoralised staff who feel the stresses and frustrations of having to work within a bureaucracy that neither understands, listens to, or has the time for local concerns.

The reports pretend that this is not the case by arguing that a single force will, through efficiencies of processes and interfaces, be able to address local needs better than a decentralised organisation could. But this is smoke and mirrors as addressing local needs would decentralise control and increase the interfaces involved in decision making processes. The force as a whole cannot be both centralised and decentralised at once.

Different operating structures favour different ways of working. Centralisation makes for efficient delivery of very simple standardised unchanging services but adds complexity and bureaucracy to the delivery of complex and highly variable services such as are required to address the diverse needs of a nation like Scotland.

Decentralisation is a potentially inefficient way of delivering simple standardised services but excels at motivating staff who are faced with dealing with ever changing diverse and complex issues at the local level.

The recent Christie Report on the delivery of public services in Scotland makes it clear that we can only operate efficiently if we accept that there are different needs and priorities across the country and that we need to address those differences. In other words, we will deliver cost savings, not by delivering the same service throughout the country but by prioritising the delivery of the services that matter in the areas that they are delivered. The simple truth is that delivering the same service everywhere is a waste of money.

Cost of change

Incredibly, the suggestion of moving to a single force for the police, fire and rescue brings one certainty: the cost of change. Whilst there will be arguments about the predicted cost, there is no-one questioning the fact that changing to a single force will be a big and costly change.

If there was a strong argument why, in the long run, a single force would bring savings and improvements in service then you could understand that this cost of change would be something worth investing in.

But the majority of the published cost savings are not dependant upon a single force. They relate to reducing capacity and lowering salaries.

In simple terms the government is proposing to reduce the workforce, cut salaries and simultaneously ask them to completely change the organisational structures they work within. If that is not a recipe for disaster, what is?

Why?

What is driving the push towards centralised control?

The only answer that seems to make any sense is centralised control itself. It seems that we have a government that does not believe in giving out control to other people. It seems to believe that our country is safer in the hands of a few hand-picked individuals in arms reach of government: but out of touch with you and me.

Investigating the figures behind the facts - The future of policing in Scotland - the one force model

2011-08-06T22:14:00.000+01:00

"The projected long-term savings under a new structure are up to £154m a year, every year." (Scottish Government Spokesman quoted in Scotsman, 26 July)

I recently read some newspaper articles about the current political debate on merging Scotland's regional police forces into a single force. One figure kept on being quoted (without reference) and grabbed my attention. The £154m annual savings that would come from this change. Where did this figure come from?

The figure appears to come from the document: "Sustainable Policing Project, Phase Two Report: Options for Reform" (March 2011). This report presents an estimate of potential savings (p.55) which totals £153.9m.

But this figure is not the potential savings from restructuring from the current regional forces into one force: the table shows, to quote: "estimated efficiency potential within the various functions of policing if the Target Operating Model were to be adopted". The "Target Operating Model" is a model for operation which is independent of whether the force is 8 forces, regional forces or one force. To quote, "The majority of these efficiencies are achievable irrespective of structure" (p. 68).

£154m is the estimated savings that could come from other changes to the way that the police forces operate. Some of these savings are quoted as potentially taking up to five years to realise. "Efficiencies were calculated and validated using benchmark data" (p. 54) "from other forces (England and Wales) public and private sector (Industry Standard)", "by considering the potential impact of established and well tested levers, for example: management de-layering, process engineering, shared service economy of scale".

The quality of this report is very poor and the savings take no account of the cost of implementing change. Cosla is quoted (by Scotsman) as saying that a Scottish Government business case, "shows that Scotland would have to spend a whopping £230m (equivalent to 7,600 police officers] in start up and restructuring costs for a single force." I haven't been able to find this document, but it is quite possible that this is just the cost of merging to a single force and not the cost of implementing the "Target Operating Model". Remember, it is the Target Operating Model that is quoted as delivering the savings, not the change to a single force.

I am not interested in politics. I am interested in effective public services.

If the SNP are quoting £154m savings as justification for a single police force in Scotland then they are either badly mislead or badly misleading.

I sincerely hope that those with the power to influence this decision get their facts straight before they make a very costly mistake.

I am disappointed that the media in Scotland seem so keen to present political dispute that they have failed in their job of uncovering the true facts behind the story.

Reactions to the Christie report on future delivery of public services in Scotland (or "why a little knowledge is a dangerous thing")

2011-07-01T00:19:00.006+01:00

Yesterday saw the publishing of the Scottish Government report "Commission on the Future Delivery of Public Services" chaired by Dr Campbell Christie. Despite presenting a radically different way of delivering public services, it has met with repeated media criticism that the report says nothing new or actionable. Damning with faint praise, the SNP Finance Minister, John Swinney, in a Newsnight television interview, indicated that the proposals were broadly in line with what the government was already doing.

So is this report 100 pages of predictable "Motherhood and apple pie" or is there some substance? If there is substance to this report, then why is the media unable to see it and why are contributors to the report uneasy to back it wholeheartedly?

Lacking substance?

It's clear why many people see this report as lacking substance. It doesn't provide specific steps on moving forward like saying that region X should merge it's 5 frontline services (1,2,3,4 & 5) into a single coordinated user facing service by May 2012.

But the reality is that this report was not commissioned to do this, nor did it have the capacity to get into that level of detail. Politicians know that the media like this kind of bite-size proposition and they are good at delivering them. We should think twice about criticising a report of this quality, depth and coverage for failing to feed the media.

The report presents and argues the case for long term focused strategies for delivering public services. These strategies may not appear radical on paper but the fundamental issue is that these are strategies that are completely different from what is happening today.

It doesn't matter that these individual proposal seem obvious. There is a reality gap between what most people see as obvious and what happens in practice every day. It is a scary truth that a large number of the people who deliver our public services are either working with systems that prevent them from doing the job they want and need to do, or are so isolated from the underlying purpose that they cannot see why they are failing to deliver value.

I shouldn't need to explain these things, because the report actually says them very well. But there is clearly a need to explain how this report will make a difference.

Making a difference

If the proposals and implications of this report are delivered then we will see a massive change in the delivery of our public services. The changes, whilst following common themes, will differ from region to region and service to service. They will be driven by the needs of the communities they serve and those needs vary dramatically across the country.

These proposals are based around service delivery and organisational change that have been tried and tested. If you need evidence then search the internet for some of the books or online video presentations of John Seddon whose company "Vanguard" has become synonymous with the practice of 'Systems thinking' in the UK public sector. Like him or loathe him, John Seddon has applied these principles throughout the UK and collected the evidence to demonstrate the results.

If you want something even more radical but closely aligned to this reports proposals of reducing organisational complexity, get your hands on the BBC Wales series, "Ban the Boss" and the work of Paul Thomas. His truly radical and fairly risky work in the public sector in Wales claims up to 300% increases in productivity. Yet, on paper his strategy of reducing "managers" could so easily be criticised as being nothing new and could, so easily be implemented in a way that delivers a disaster rather than reduced cost and improved service.

The problem

The problem is that most people who read this report have no idea how to implement the changes. These changes might as well be written in a foreign language or complex scientific notation.

Despite being in English, the language of much of the proposals of this report is the language of organisations, systems, complexity and change. This language draws us in to a false sense of understanding with terms that seem to make sense yet hold vast meaning beyond common usage.

Page seven of the report makes cursory reference to "failure demand" which is explained as "demand which could have been avoided by earlier preventative measures." That sounds so obvious and yet the topic of 'failure demand' is the subject of whole chapters if not whole books. It is neither an obvious concept nor one that is well understood or applied either in the public or private sector.

In other words, this report contains big ideas that are compressed into succinct and specialist language that should be fully understood by those who need to implement the proposals but could so easily be missed by the general public or journalists or indeed politicians.

Actions

So, how can these ideas be turned into action? The secret sounds simple. They need to be implemented by people who understand the ideas.

Changing organisations is not something everyone gets taught about at secondary school. Even those with management degrees will be lucky to have received any guidance at all on how to restructure organisations to reduce complexity. Yet, unless something radical happens, this report will be digested and actioned by people who have no specialist knowledge or experience in organisational change and systems thinking.

The Christie report is a radical report. But it's proposals will have little effect unless the government can find and appoint a large enough group of skilled individuals who have the capacity to assess regional issues and turn them into practical proposals. Those individuals need to be given the authority to ask questions and initiate change.

Conclusions

In summary:

There is no point in trying to turn this into a standardised and centrally driven policy of change across Scotland. It needs to be driven at a local level.
It has to be implemented by individuals who have been selected because they fully understand the organisational issues being presented not because they are "solid and reliable people who have delivered good reports in the past". This also excludes appointing a group of "management consultants" from some "safe" big brand consultancy.
By delivering these policies locally the government benefits from spreading its risk. Not every plan will go according to plan. Some of the people chosen to drive change locally will not live up to expectations. But only fudging the facts or avoiding the issues will avoid this and the cost of doing nothing is far, far too high.
If the government acted quickly they could have a handful of specialists in place by the end of August 2011. Acting locally with central backing they could provide the skills and energy to start delivering the first steps of change immediately.

Postscript

There is clearly a large overlap in Scope between the Christie report and the last week's McClelland report on the ICT Infrastructure of the Public Sector. If the government is going to deliver joined up services then the proposals of these reports should be joined up too.

Response to Strategic Principles of McClelland Report "Review of ICT Infrastructure in the Public Sector in Scotland"

2011-06-24T15:07:00.004+01:00

Apologies in advance for the length of this post. The post consists of the 12 proposed strategic principles of the McClelland Report with comments after each one.

Direct comments and responses to the 12 Proposed Strategic Principles of the "Review of ICT Infrastructure in the Public Sector in Scotland":

"Although 'information management' is a core activity it is not essential to operate totally self-sufficient local information processing, support and development."

This statement on its own is irrefutable and I would be surprised if any public sector body was truly "totally self-sufficient". However, the implication of this 'strategic principal' within the McClelland report strongly argues the case against local 'in-house' knowledge. And the reduction of local in-house knowledge is a dangerous proposition that does need to be evaluated very seriously in each context.

In short, removal of or outsourcing of local ICT knowledge disempowers the local body from making choices or innovating around the use of technology and process. Given that innovation efficiencies come from local users this is paradoxically a recipe for inefficiency and waste.

At the end of the day skills and knowledge need to be located as close to where they are needed as possible. If local bodies need local skills to make efficient use of ICT and process then the skills need to be there.

"The number of data centres and associated support should be minimised."

Yes.

"The shared hosting of common applications delivers ICT savings and central and regional plans open up ICT and other shared business process service opportunities. The existing clusters of nearly common applications should be built upon by selecting the best single application implementation and associated business processes and then from there achieve a reconciling of and agreement on common business processes so that the number of separately hosted instances can be rationalised and reduced."

The first sentence here is fine and good. Shared hosting of common applications is good and coordination at the regional level to encourage or even mandate the use of shared common applications may well be the right thing to do. Moving staff from using Microsoft Office to using OpenOffice.org would be a classic example of appropriate mandated change as long as all genuine technical data exchange issues had been addressed in advance. Of course such a change has a change management cost which must be costed.

However, as soon as there is talk of trying to merge 'nearly common' applications or business processes the line has almost certainly gone too far.

It is true that organisations may have different processes and procedures for historical reasons and there may be no reason for them to be different. If the end users who are familiar with everyday use of such systems recognise this, even reluctantly, then there is probably a strong argument for merging the processes as long as the costings recognise that there may be a significant 'investment' cost associated with implementing that change process.

Unfortunately one of the commonest reasons for so-called IT disasters in the public sector is the attempt to merge processes that seem (at a distance) to be the same, but are (on close examination) different for good reason. Often the historical reason why different organisations do things differently is that there is something genuinely different about what they do!

Changes in processes are often introduced locally as innovations to reduce cost and increase efficiency. Good reasons why changes in processes can bring local efficiencies include: co-location with other services; the shape of buildings and how long it takes to carry out tasks; office design; shared vs person printers; shared vs personal terminals/computers; and security implications of public access to buildings on logon procedures.

There is nothing worse on staff moral or fundamentally stupid than forcing staff to change a locally efficient process to an inefficient one in order to conform to a standard way of working!

"A framework of oversight and governance for each part of the public sector and at an overarching national level is critical."

Yes, but it should not add layers of bureaucracy or oversee ICT in isolation from other issues which are always interconnected.

However, there must be mechanisms in place to allow innovation from the users. In other words, there must be an open communication channel from users to those who can instigate change.

"Co-ordination of interaction with the ICT industry within each sector and at a national level is essential and will be beneficial."

Yes. There is opportunity for procurement savings by ensuring co-ordinated interaction with suppliers and service providers. It is essential that public bodies have skilled ICT representatives involved in those interactions to ensure that best value is being obtained. There may be a case for involving SMEs as independent ICT advisors in these situations as SMEs tend to be more aware of technological change than public sector ICT departments or large private sector suppliers.

"Relationships with suppliers should have a stronger partnership element."

What does this mean? Many historical suggestions of strong partnerships with the private sector have become costly and anti-competitive relationships which have benefitted the suppliers at the expense of the public sector.

The public sector needs to establish relationships with suppliers that leave the buyer with ongoing control and choice. One of the best ways of doing this would be to ensure that all software development for the public-sector was based on open standards, publicly owned and open-sourced from day one. This would put the supplier under appropriate public scrutiny and enable a poorly performing supplier to be swapped out in favour of an alternative supplier if required.

"It should not be a given that investment in and ownership of ICT assets and capability such as systems development is the norm and all avenues including investment avoidance and transaction / usage based charging should be pursued."

Scotland has a particularly poor history of choosing to do its own thing when something perfectly good is already available to purchase 'off the shelf'. This 'not invented here' syndrome is something that needs to be rectified particularly in the primary and secondary education sector where recognition of Scotland's different education system has to be balanced with a recognition of its similarities.

The rapid growth in cloud based services or 'apps' also means that there are large areas of ICT usage which could be satisfied by 'transaction / usage based charging' models.

However, where systems are developed specifically for the public sector it is very dangerous to do so under an agreement that gives ownership to the supplier and leaves the public paying on a transaction / usage basis. Any system developed specifically for the public sector should be owned by the public.

"Citizen services and data should be seamless and integrated across public sector and should specially address the needs of the elderly, sick and other vulnerable groups which cross organisational boundaries."

It shouldn't be necessary to warn of the mistakes of projects as recent as the NHS National Project for IT. But history shows that the public sector has repeated these mistakes over and over again.

We must not embark on producing any "seamlessly integrated" system across the public sector.

The recognised way to address this need is to establish well defined data standards and protocols for transacting data between systems. Wherever possible these standards should be open standards because we should not be re-inventing them and we should be trying to benefit from re-use of systems that already exist.

Thankfully the NHS in Scotland has largely taken this approach already.

"Most of the required ICT capability is specialised by sector yet there are vital national dimensions and cross-sector imperatives."

Yes and open standards for data sharing should be a big part of the mechanism for addressing this.

"Order of merit should be to first re-use, then buy and build only as a last resort. Existing initiatives and exemplars should be built upon and have their capabilities extended through free sharing with others."

Free sharing could be transformational on the public sector. But this should, wherever possible, be public sharing not just sharing privately between organisations.

Firstly, unless software used by the public sector is made publicly visible and accessible, organisations will find it hard, if not impossible to identify the opportunities for re-use.

Secondly, only through public sharing will the public sector benefit from the untapped resources of SMEs and crowd sourced public individuals.

Open sourcing public sector software massively lowers the barriers for competition from SMEs as it enables them to compete with incumbent providers and enables them to pro-actively offer services tailored specifically to the technical needs of the public sector.

"New technologies and concepts be pursued especially where they can reduce investment and support other efficiency and sustainability goals."

This is a good goal but the original 'strategies' above would all but prevent innovation and the use of new technologies.

"The negative impacts of and positive opportunities from effective ICT on the environment should be addressed and pursued."

Thankfully a focus on cost reduction in ICT tends to naturally deliver environmentally beneficial outcomes.

Shared ICT services are a recipe for holding back change

2011-06-22T00:58:00.001+01:00

Today I've been reading John McClelland's report, "Review of ICT Infrastructure in the Public Sector in Scotland". Published by the Scottish government today, the work was started in 2010 at the request of the Scottish Government Cabinet Secretary for Finance and Sustainable Growth.

There's plenty to comment about in this report but I'll focus very briefly on one important point. But first a quote from the report:

"Shared deployment of ICT will reduce ICT cost and deliver savings in costs within individual public sector bodies. However, it can also, by being shared, provide a platform for additional efficiency and savings across multiple public bodies. The establishment of shared hosted information systems, commonly used across multiple organisations makes it very much easier to also share the resources and skills needed to operate other business processes. In this way shared ICT deployment unlocks the gate to shared services opportunities in other operations and processes." (section 4.2)

It is true that the sharing of flexible and completely industry standard infrastructure can help to reduce costs and reduce duplication.

Beyond that, organisations are not the same. So the sharing of services is not about sharing the same thing it's about making organisations work the same way whether it makes sense at the local level or it does not. The cost of making this massive organisational change may completely outweigh any apparent short term savings in ICT procurement. The long term process inefficiencies of forcing different organisations to work in the same way may introduce costs that also outweigh any ICT maintenance savings moving forward.

In short, there can be no assumption that shared deployment of ICT will reduce ICT cost.

The biggest problem

But the biggest problem with shared services seems to be the least understood:

Sharing services in the ways commonly understood in the public sector, lead to tight process dependencies between already huge government organisations. These dependencies, instead of helping to deliver change, actually act to constrain future change.

To put it simply. If you force two organisations to use the same core processes and the same central ICT system, then any future changes in process must be implemented in both organisations at the same time.

Government IT - central government and misunderstanding efficiencies of scale

2011-04-01T11:17:00.000+01:00

Last week, the Commons Select Committee on the Use of IT interviewed Local and Central Government departments and the largest supplier to government which also was the only large supplier willing to take part as a ‘public’ witness. This week they questioned the government minister but I haven’t got round to watching that yet as the video stream requires me to watch in real time and be physically in front of a computer with the appropriate plugins installed. I would have preferred to be able to watch it at double time on the bus!

The local government representatives, though they may not be representative, largely demonstrated an understanding and engagement with issues even if, in one case, they found it hard to express that in ways that could be understood by the committee.

Central government departments

The representation from the central government departments gave me considerable cause for concern. If I were responsible for their current projects I would be asking lots of questions. Here are a number of the comments that rang warning bells:

60% of the current project is being done in an agile way, the other 40% (core infrastructure) is not.
“legacy systems are not suitable for agile”
The advantage of using an existing supplier is that you can re-use skills and people and existing knowledge.

For clarity, I should explain why these rang warning bells:

One of the key benefits of agile is the ability to deliver a very early release of a product ‘end-to-end’ within a short time-scale. The act of doing this de-risks a project immensely and the act of failing to do this is a much less expensive way of discovering that something is wrong! You can’t deliver an end-to-end early release if the core infrastructure is not part of the system. So, there may be apparently valid reasons why this project cannot be 100% agile, but those reasons, or the decisions made in response are a sign of a problem.
It simply isn’t true that legacy systems aren’t suited to agile management / development techniques. In many respects, renewing (‘refactoring’) legacy systems is a task that benefits hugely from the principles of agile. In particular, agile provides skills and techniques for reducing the cost of change and one of the big problems with legacy systems is their cost of change. So, this quote, from one of the key leaders in the public sector, tells me that they simply do not understand what ‘agile’ means or can do.
Continuity is clearly a great way of retaining knowledge and skill. I think government should be doing a lot more to retain skill and I think procurement practices tend to enforce the throwing away of knowledge and skill, ultimately at the expense of the public who pay for everything. But you cannot, on the one hand claim to be running a procurement process that is even handed and at the same time argue that re-using the same supplier enables you to re-use skills, people and existing knowledge.

Integration, centralisation and efficiencies of scale

Anyway! All of these evidence sessions to date have thrown up enough material to write a book which is why it’s taken me another week to write anything coherent.

But, I’ve been saved because this week I reached page 57 of the “Written Evidence” document - the submission of Andrew Hardie which very effectively expresses my views on almost everything. To be honest, this is quite disconcerting at first, but also very encouraging as writing has never been my strong-point! To quote one of his points which I agree with wholeheartedly but which is a point made by few if any of the other contributers:

Perhaps, the greatest fallacy in both government and private sector ICT systems implementation is that greater systems integration is the answer. It isn’t. The more tightly you couple ICT (or, indeed, any) systems together, the greater the speed, range and impact of problems and side-effects become and the harder it is to change the resulting monolithic systems, precisely at the time when ever greater agility is needed.

There is a constant push to standardise systems in government to obtain ‘efficiencies of scale’ through more efficient procurement or through the wide use of identical software. This push for standardisation and efficiency almost always results in a decision to roll out a massive IT project. The conversation appears to go something like this:

A: We mustn’t have any more huge IT projects that cost too much and go wrong.

B: I totally agree.

A: Okay, so how are we going to reduce costs in IT projects.

B: Well, we are aware that there are lots of departments re-inventing the wheel with different IT systems that do fundamentally the same thing. We need to standardise the systems they are using and cut the outrageous costs of this duplication which is caused principally by a failure of communication between departments. The left hand doesn’t know what the right hand is doing!

A: Yes, I totally agree. It’s shocking!

B: Okay, so lets set up a project to consolidate all the systems together into one standard system.

A: Fine. Go ahead!

Yes, in the course of a few sentences they have moved from deciding not to have a huge IT project to setting up a new one, yet the flaw in the argument seems hard for government to spot!

There are all sorts of costs of complexity associated with interactions with people. Very rarely do multiple departments actually do the same thing. When you try to bring everything they do together into one system you typically end up with a system that is far too complex or a system that doesn’t meet the needs of the people you are producing it for.

The fundamental flaw in the argument above is the belief that different departments are actually doing precisely the same thing. In reality, if you talk to the users, you will find lots of differences in need and use and common practice. If you want to make the users more efficient at doing their work then you need the system to be tailored to their need not imposed on them from above.

The more complex flaw is that they criticise the departments for not communicating without recognising the additional cost and complexity introduced by requiring departments to communicate with each other! Failure to communicate is a cheap argument that is almost guaranteed to be accepted as grounds for criticism. But communication takes time and we almost universally suffer from too many meetings and not enough doing.

Government IT projects and managing risk

2011-03-18T10:46:00.002Z

I’ve been trying to watch / listen to the witness sessions of the Commons Select Committee on the “Use of IT” that have been taking place over the last two weeks. The strongest theme that I detect in these sessions is the bemusement that past governments and past projects have failed to address the issues.

One word that keeps on coming up in the discussions is ‘agile’. As always there are many explanations of what it is and I can’t help but feel that none of the explanations seem to provide the select committee with the eureka moment that people need to understand what it means and why it is different from the status quo. I wonder whether an explanation in terms of the management of risk might help a little.

Agile project management and the management of risk

For the sake of understanding what agile means, particularly from the perspective of management and strategy, let’s consider IT projects and the management of risk.

There is a strong tradition in some areas of management that says that management is about control. In other words, a well managed project is a project that is well defined, fully costed and running to plan. If a project fails, then the response is that the project was not controlled enough; it was not properly defined at the beginning; it was not properly monitored throughout; it was poorly costed and lacked strong management.

This view of management has been the driving force for many standards and methodologies that government has adopted over the years to keep projects in check and ‘ensure value for money’. But as we all know, history shows that, in a lot of cases, this simply hasn’t worked.

In reality, so called government ‘IT projects’ are often massive ‘change projects with an IT element’. These projects are massively complex and inherently highly risky. Just like risk in the financial markets that risk can be managed, but however it is packaged up and hidden away it will not go away.

Let’s take this analogy of financial investment a little further. Imagine that you are given some money to invest in equity in an area of the market that is highly risky but has the prospect of significant returns. Do you:

Invest all you money in one company, locking yourself into an agreement that prevents you from selling your investment for five years; or
Spread your risk across a number of companies, ensuring that you can sell your investment at any time and re-invest it somewhere else if the market changes or you realise that you made the wrong decision in the first place.

Now, imagine that, for some bizarre reason, you take the first option and you invest everything in one company. Would you reduce your risk if you took two years to decide which company to invest in and you ask each company to set aside a considerable period of time and resource in making detailed projections of where the company would be in five years time?

Okay, I hope it’s clear to see that this analogy bears remarkable resemblance to the traditional government approach to contracting out IT projects. I also hope it is clear that, when seen as a task of managing risk, the second option is clearly the way to go.

The second option is, in effect, the ‘agile’ approach. You diversify your risk and you reduce your risk by simplifying a big problem into a large number of smaller problems. You push against those who say that a project is inherently large and find ways of building a small working part of it very quickly. You push against those who say that everything needs to be produced by one company or team and use existing open standards as a way of guaranteeing interoperability. In this way you prevent ‘lock-in’, you open up projects to a much larger number of smaller development teams, and you encourage cooperation, experimentation and innovation.

I hope, for a few people, this explanation might help to bring a better understanding of some of the problems of government IT projects and why an open and ‘iterative’ (under whatever name) approach is crucial if government is to manage its risks and deliver the real potent for a huge return.

Written evidence to the Commons Select Committee on the Use of IT

2011-03-11T10:25:00.001Z

A number of recent articles have drawn my attention to the Commons Select Committee on the “Use of IT” which apparently started an inquiry today, but then again the web site said that yesterday too!

I’ve always had a very keen interest in the effective use of technology in the public sector, so I couldn’t help take a look at the subjects it was looking into and the written evidence it has received. Unfortunately, so far (and I’m only on page 15!), the evidence seems to reinforce the reasons why successive Governments seem to be so fundamentally unable to use IT effectively. The reason I say this, is that the evidence I have read so far demonstrates the massive diversity and contradictory nature of the advice they receive.

In this first blog on the subject I’ll touch on the suitability of management and delegation out of the public sector. I’m fairly sure there will be more posts to come on this who subject!

Suitable management

One theme that is already coming through in the first 5 submissions is the theme of control. Are the management suitably qualified to manage the projects:

Management of the Public bodies is at all levels recruited mostly from non-IT backgrounds. Typically the managers possess long experience of the needs of the Organisation, but not of the issues raised by the development of an IT system. This leads them to make errors of judgement on IT policy.

I think the background of the management is probably a bit of a red herring, but it is absolutely true that a manager of an IT project must posses the skills and knowledge to be able to manage the project and take control. A common characteristic of management of failing IT projects is the complete reliance they have on the guidance and work of staff and consultants below them. The best manager of an IT project is the manager who can, if she needs to, listen and talk knowledgeably and meaningfully with the ‘shop floor’ IT staff doing the work and with ‘shop floor’ public sector staff who trying to use it. This is not about a public relations exercise of having top management ‘come down and talk to the workers’ this is fundamentally about whether the manager in charge has the knowledge to understand the problem and talk the language of the people who are implementing it both on the side of the technology and on the side of operations and change management.

Delegation

Closely related to this issue of management is the issue of where you place responsibility.

There are many politically motivated reasons for deciding whether public projects should be carried out within government or in the private sector or even in the voluntary sector. However, ultimately whilst this can have a significant impact on the way in which the work is carried, it doesn’t address the fundamental issues of why IT projects succeed or fail.

This quote from one of the submissions is a perfect example of this misunderstanding:

Government already has a model for a highly successful, cost free, technology implementation that has reached 80% of the UK population. It is the National Lottery. The technology is complex, secure and costly. It cost the taxpayer nothing because government intelligently pulled the levers that shaped an opportunity the private sector would fund.

Here the success of this project is attributed to the government distancing itself both managerially and financially from the risks of implementing an IT project. But in reality this project possess very few of the fundamentally difficult characteristics which government normally has to deal with in IT projects:

This was nothing new. The UK was implementing a National Lottery on the heals of many other nations who had done this before. There are few areas of public sector IT which fit so neatly
This was a perfect example of the large scale delivery of a standardised product. The National Lottery was not an IT project to modernise hundreds if not thousands of previous lotteries scattered around the country and carried out in their own unique ways in circumstances which were also very different regionally. The National Lottery was a blank sheet of paper with the opportunity to implement the best, cheapest, most effective solution and to effectively impose it consistently across the country.
The requirements were relatively easy to define. Ongoing success was easy to measure. Ongoing success and incentives were aligned with ongoing government requirements.

In short, the current government may well have it’s own political view on who should carry out IT projects, but this decision should not be confused with the real issues of making projects work. If the government decides to pass responsibility to the private sector or even the voluntary (should we call it ‘open source’) sector, it will nevertheless remain responsible for the end result. Ultimately the Government must either take control or loose control.

"show your working"—a plea for Scala coding quality

2010-08-31T10:45:00.004+01:00

The recent debate about the complexity of Scala over Java and other languages has lead me to bemoan the quality of some significant chunks of Scala code currently circulating in the public domain.

I probably should just be patient as the cycle of programming language maturity is a common one. In the early days of a language, in the absence of coding standards and common practice, people grab at the new features of a language and enthusiastically use them with little awareness of the consequences. As a language matures, the community of developers adopts and polices standards to the extent that a newby developer soon finds themselves instructed on the basics of what to do or not to do.

There's a lot of Scala code out there that feels like it's been put together by spotty adolescents with too many hormones and not enough common sense. Okay, I'm being harsh, I'm a Scala newby too, but I think it really is time we started to get a bit more focused on developing some good coding practice and mentoring each other towards maturity.

I'm going to start with a simple issue which is close to the recent complexity debate...

Extremes of pure functional style encourage code obscurity

Functional programming is great, but taken to an extreme functional programming can:

encourage short meaningless function names.
discourage temporary named variables which help to explain what is going on.
encourage the creation of large numbers of small functions which obscure the modular interfaces.

Take this function as a slightly extreme example:

  def ^!!||^(implicit f: Foldable[IN]) :
      Kleisli[Option, String, NonEmptyList[List[Char]]] =
          kleisli((p : String) => {(this !!|| p).toNel })

I'm not trying to pick on anyone here, so apologies to the author, but this method is one of over 50 public methods in a single Scala trait of an open source Scala library. Is it any surprise that people think Scala is complicated?

So, here are my Scala programming tips for the day:

Minimise your modular interface: in other words, keep your publicly accessible functions, properties and classes, etc. to a minimum. If it doesn't need to be public make it private. Why? ... because developers reading your code shouldn't be left swimming around trying to find what's important and what isn't. If you create a function that is only used by one other function, define it within the function that requires it. This makes it absolutely clear that it is only there to support the other.
Use meaningful names and avoid symbols: if your readers can't figure out what a method is supposed to do then chances are neither will you in a few months time! The use of symbols for function names is a curse of the current trend in DSLs (Domain Specific Languages). Yes, do define a + method to add two objects together, but don't write functions called, ~>, ~~> and ~%> and think you're clever!
Don't be afraid to use vals to store partial results in your function rather than trying to string everything together into one long expression. Not just does this simplify the code by breaking it down into smaller parts, it also provides you with the opportunity to "show your working" by assigning partial results to values with meaningful names. I doubt an assignment to a val has any impact on performance. But I'll leave it someone more qualified in bytecode to back me up on that.

Okay. That's me done for today. Don't miss out on this great Scala community style guide. A really valuable contribution to Scala maturity.

The official way to bypass data modification on O2 mobile networks

2010-08-16T10:27:00.001+01:00

Last week I received details from O2 of the official way of bypassing the 'optimisation platform' that O2 use on their mobile networks. They were particularly concerned that I pass on the following comment on the use of this and so I include it below verbatim:

when using this “bypass” function you must consider page impressions will take longer, on average, and thereby detract from the user experience, a slower experience. Also, a greater volume of data will be downloaded , on average, and those customers who do not have unlimited data will use their data bundle faster or incur high bills.

They then then went on to reference section 14.9.5 of the W3C HTTP 1.1 Protocol specification: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.5 which refers to the use of the HTTP Header "Cache-Control: no-transform" response directive. By setting up your server to return this response header, O2 indicate that they will not modify the data.

I should add that I haven't had a chance to test this yet and in particular, to test whether this stops the compression of images as well as the modification of HTTP source code.

Whilst this appears to offer a way for web developers to prevent their site content from being modified, it does not resolve issues for developers of web applications which utilise web data feeds that are not under their own control. For example, the developer of the great iPad Viewfinder app which provides Flickr photo search and download cannot prevent O2 reducing the quality of the images when using it over their mobile network.

O2's response to data modification queries

2010-08-16T10:03:00.001+01:00

Just over a week ago I had a conference call with Tim Fielden (Data Access) and Gavin Sheldon (Head of Networks O2 UK).

They indicated that they had looked into the legal issues I had raised and were advised that the data modifications they are making do not cause web site developers to infringe any copyright or licensing terms. As I said to them at the time, I still find it hard to believe that O2 themselves can modify other peoples data without informing them.
They recognised a need for developers to be able to bypass their data modifications in more exceptional circumstances and said that they would send details of how to do this. I'll explain this next in a separate entry so it's easier to search for in future.
They expressed confidence in their extensive testing of their network and data compression/modification systems and said that, in their testing, they found that these network optimisations resulted in the best customer experience. I specifically queried whether or not they tested accessing multiple pages in a row to which they reiterated their confidence in the extensiveness of their testing, but welcomed any specific cased of web sites that were found to be underperforming on their network.

All in all they were very polite and clearly expressed their desire to be helpful but I was left feeling that it was going take more than a few customer complaints before they would do anything to stop modifying data without telling anyone. No-doubt they feel that that, as not the only operator to be doing this in Europe, they don't need to rush to change the status quo.

Operators modifying mobile data content (a table)

2010-08-03T14:04:00.006+01:00

Following my request for people to test their own mobile provider, here is an up-to-date summary of responses so far. The number under "Modifies" or "Doesn't appear to" is the number of individual confirmations I've received. I won't update this after I've received 4 consistent responses.

Please note that "modifies" means that the simple javascript test appears to show that the provider is inlining CSS style sheets. The "Doesn't appear to" means that the test didn't detect CSS inlining, but doesn't exclude the possibility that the operator is still modifying content in a different way.

Country	Operator	Modifies	Doesn't appear to
Australia	Optus	-	1
Canada	Fido	-	1
Germany	Vodafone	2	-
Netherlands	T-Mobile	*	* (Note 1)
UK	O2	4	-
UK	Orange	-	1
UK	T-Mobile	2	-
UK	Three	-	2
UK	Vodafone	-	1
USA	AT&T	-	1
USA	Speakeasy	-	1

T-Mobile in the Netherlands apparently gives users a choice when they subscribe to the service.

O2 Data Modification Update

2010-08-02T15:24:00.001+01:00

A number of people have asked for an update on the O2 data modification situation. Here is a quick list in no particular order:

The Head of O2 Networks has responded to my communications and has arranged a meeting later this week to discuss the issues. He has also asked the O2 legal team to reassess the licensing implications of the modification of licensed source code including the removal of copyright messages.
The data modification technology used by O2 appears to be provided by the company ByteMobile which appears to suggest that some of its technologies are also in use by Orange, Vodafone and T-Mobile in the UK.
A 'back of an envelope' calculation appears to indicate that the data modification taking place will indeed significantly increase the amount of data being sent to your mobile device when you access two or more pages of a standard web site like the BBC news or Guardian news web sites. This is on the assumption that your mobile device would normally locally cache CSS and JavaScript files. It's quite possible that some older mobile devices do not do this.
The Register (online technology publication) has responded to me about this issue but to date I don't believe that this issue has been published in any press article. I think people generally find it a little too technical to write about. Everyone I have spoken to has been completely unaware that this practice is going on.
A comment making reference to this issue that I added to a BBC news article was removed as being "potentially defamatory".
Following issues arising with another web site, this blog has followed up on the same issue: O2 UK mobile users - your operator is breaking this site for you..
StartupCafe kindly posted an article on my investigations: O2 Compression on mobile devices.
I have contacted Ofcom who are assessing my complaint.

Is your mobile provider modifying your web pages? Instant test.

2010-07-28T16:11:00.001+01:00

Prompted by an email earlier today I've produced a very simple web page that carries out a very simple test that checks whether one of the modifications to web pages has been carried out when the page is loaded.

If you access the page Inlining Test on your mobile device whilst connected through your mobile network (not through WiFi), you should see one of two messages:

"No identified modification has happened to this page"; or
"This page has been altered! It appears that the stylesheet on this page has been inlined by your network provider."

It would be really helpful to get some feedback on which providers, mobile devices and APNs are exhibiting issues and in which parts of the world. If you could leave a comment if you have a problem on a setup that hasn't already been listed then that would be great.

Please note that this test hasn't been extensively tested so I can't guarantee it will work on every mobile browser. Feel free to let me know if you spot a gaff!

For those interested to know the page contains a simple JavaScript that checks to see that the stylesheet link on the page is still there once the page has completed loading. If it isn't there (most likely because of the Bytemobile filter inlining the JavaScript) it displays the message to say that. This page also has Google analytics on it so that I can get a feel for how many times people are using it.

Letter to Ronan Dunne, CEO of O2

2010-07-23T16:15:00.000+01:00

Dated 23 July 2010:

Thank you for your response.

The devices I have personally used to confirm that data modification is taking place are an iPhone and iPad.  However, I have received confirmation from other users that that modifications are taking place for mobile broadband users using laptops as well, and I know that many of the practices are also occurring for users with other mobile phone devices.

In specific reference to the UK Government web site.  The UK government web site uses the SWFObject JavaScript library which is made available under an MIT license which requires that the license is distributed with the software.  It uses the jQuery JavaScript library 
which is also distributed under a joint MIT and GPL Version 2 license. I believe the GPL license prohibits the software being combined with any other software or modified.  There are other works also used by the UK Government web site.

Currently, as far as I can tell, the O2 mobile network is running a caching proxy server which is combining JavaScript libraries into the main HTML source of web pages and removing copyright messages.  I believe, and the informal advice I have received would confirm that this action is in contravention of the licensing terms.  As it is O2 that are doing this then I believe that O2 is breaking the law.  I should also make clear that I referred to the UK Government web site to try to ensure that my email was not ignored.  These practices are being applied to every web site as far as I can tell, so there will be literally millions of other examples.

However, the broader and arguably more serious issue is the fact that O2 has a practice of modifying any data passing through their network between senders and receivers without notifying either party of the changes that are being made.  Even when I tried to find out what was happening I was told by your technical department that they *did* compress data but that they were not legally required to give any detail of what they did. (I should add that this statement was misleading as they do a lot more than just compress data, they also change and remove data).  In other words, even if I ask, I am not told what O2 are doing to my data.

I believe that this practice may nullify the contractual obligations of every customer of O2's mobile data services at the present time.

Finally, I refer you to European legislation being formed around 'net neutrality' <http://europa.eu/rapid/pressReleasesAction.do?reference=SPEECH/10/153> and curently being considered by Ofcom.  The Vice President of the European Commission said: "Transparency is non-negotiable. This is already addressed in the new regulatory framework, but the principle is worth re-stating: in a complex system like the internet, it must be crystal clear what the practices of operators controlling the network mean for all users, including consumers."

Kind regards,

Stuart Roebuck.

Correspondence with O2 about their data compression policy

2010-07-22T10:31:00.004+01:00

Here is O2's response (22 July 2010 emphasis mine) to my enquiry asking for an explanation of their policy regarding modifying data sent over their mobile network. For those readers who are not clear on the technical details, the blanket statement that any phone compresses the data for better viewing is not true.

Hello Stuart

Thanks for your email about the data compression on your iPhone.

I get to know from your email that you wish to know our data compression
policy.

Just to let you know Stuart, that if you are accessing the internet on 
any phone, the phone automatically compresses the data to allow better 
viewing of the websites.

This is a technical query and we've a separate team which is specially 
trained on all the technical issues. So that you can get a satisfactory 
response to your query, all you need to do is speak to this technical 
team on 08448 752 302. We're open: 

- Monday to Friday between 8am and 9pm. 
- Saturday between 8am to 8pm
- Sunday 8am to 6pm.

Calls to the above number are free from all O2 Pay Monthly mobile 
phones. You can also dial 2302 free from an iPhone to speak to our 
iPhone customer service team.

I wish you a great day ahead. 

Kind regards

------
O2 Customer Service

Telefónica O2 UK Limited, Registered in England No 1743099. Registered 
Office: 260 Bath Road, Slough, Berkshire SL1 4DX.

I followed up on the number they gave but reached another customer support staff member who did not know what a "data compression policy" was and cut me off when they said they'd pass me on to someone who might.

I tried again and this time the person told me that I hadn't got through to the "technical team" and that the number I had been given was "just a generic number". He said he didn't know anything about the issue and nor did his manager but he would try and put me through to a different department that might.

He then came back again and told me that he could only pass me any further information if I confirmed that I was an O2 customer and gave my account details.

Finally, after approx. 2 hours on the phone I was told that O2 could official confirm that they "compress data" and they were not legally required to give any further information about what they do.

I'm now awaiting a response from O2's CEO.

Compiled list of web references to mobile compression - mostly O2

2010-07-22T09:42:00.006+01:00

O2 (UK)

O2 discussion forum: "Disabling image compression on PAYG"
Apple discussion board: "Is O2 using a proxy and compressing images?"
O2 discussion forum: "iPad unable to bypass image compression"
Apple discussion board: "Can I improve the image quality in Safari?"
Hello! I'm Ash Matadeen blog: "Get rid of image compression on O2's network "
kcjh's world blog: "Removing O2's Image Compression on Mobile Web - Handy for iPhone users!"
EdAndersen.com blog: "iPhone O2 – how to fix the image compression"
The Mobile Broadband User blog: "O2 mobile broadband compression continued"
The Mobile Broadband User blog: "Fix for image compression on O2 mobile broadband"
The Mobile Broadband User blog: "O2 delivers Internet via mobile broadband – but what Internet?"
ByteMobile: O2 UK Enhances Native Browsing Capabilities of Apple iPhone with Bytemobile Web Optimization
What's up on APRS.FI blog: O2 UK mobile users - your operator is breaking this site for you.

Note that all references to being able to use the "APN" username "bypass" on the O2 network to bypass image compression no longer apply for iPhone users. O2 appears to perform image compression and data modification regardless of this setting.

Vodafone (UK)

3G Mobile Phone Discussion forum (UK Only): "Image compression?"

Should mobile operators be free to modify content they deliver?

2010-07-21T15:05:00.000+01:00

My recent problems with delivering a web service to customers of the O2 network in the UK has got me thinking a lot about what we expect from mobile network providers.

I was trying to think of an analogy with something we are more used to and realised that web content delivery is very much like parcel delivery: we ask for something to be sent to us, they put it in a parcel and a delivery company are responsible for delivering it to us as quickly as possible without damaging it on route. In the internet context, we ask for a web page and we rely on the internet to deliver the page to us without modification.

However, it appears that we have fairly silently allowed the mobile networks to change this arrangement. For some time now we have been, wittingly or unwittingly, allowing many of the mobile networks (O2, T-Mobile, Virgin, Vodafone to name a few in the UK) to reduce the quality of images that we download. This seems fairly harmless and well intended. If it enables us to surf the internet quicker on our mobile phones when we can't see the images well anyway, then surely this is a good thing?

But it's not as simple as that.

What if I want to send someone an image over a mobile network and the image quality really matters. Firstly, I probably won't realise that O2 are going to modify it on route and may discover too late that the lower quality image has gone to print. Secondly, even if I do realise, how do I make sure that the important image isn't compressed? There is no option to bypass this. There is no way of getting hold of the content we have requested. O2 have substituted what we asked for with something inferior and they do it on purpose without asking us or giving us a choice to opt out.

It really calls into question the whole industry of web services. If you pay a web provider for some data and your mobile operator substitutes something different as it passes through their network, who has failed to deliver? If you pay for Flickr's Pro service and use it on your iPad, can you complain to them if the images you view are substandard? Will Flickr have any idea that the images they are sending you are not the ones you get?

And it's not just about images and image quality.

O2 may be going further than other providers, I really don't know. But I do know that they are modifying the very source code of the web pages you access over the internet. So the web services that painstakingly produced and tested by companies around the world are being modified by O2 in such a way that they may not work when they reach your mobile device or 3G connected laptop. You might blame your mobile phone or iPad whilst totally unaware that the problem is actually because O2 have changed the content enough to break it.

Can web content providers test all their services through every mobile network in the world to make sure that you get the service you have paid for? Of course not!

Should you, as a customer, have to figure out whether the problems you are having come from your mobile provider or the original content provider? Of course not!

Mobile phone networks should be reliable and they should provide the content you ask for without modification. If they want to optimise their networks then they can give you the option of accepting a modified network, but it should always be possible to turn off the modifications quickly and easily if anything goes wrong.

The craziest thing about this story is that, as far as I can work it out, O2's optimisations are probably increasing the load on their network and increasing the latency of their network. I have a suspicion that if they turned it all off they would save themselves some money overnight!

I have engineered a partial workaround for the web site I host and it now works on the O2 network and works quicker than it did before!

Mobile Proxy Cache content modification by O2

2010-07-20T16:55:00.004+01:00

If you connect to the web on a mobile Safari device such as an iPhone or iPad using 3G over O2's network in the UK, then you may not realise that what you are seeing has been modified by O2 between the web site and your phone. If you are the author of a web site and find that some customers are not seeing what they should you may be frustrated to discover that O2 have changed the source code before it gets to your customer and that you have little way of finding out what they have done.

I'm sure O2 is doing this in order to speed up the service and reduce network load, but unfortunately their modifications are not without consequences, as I discovered trying to host a site designed specifically to work well on an iPad.

Image reduction

The most widely recognised change they make is to reduce the resolution of image files. This has caused some complaint recently from iPad owners who have a large enough display to easily spot the different in quality that results when, for example, browsing photographs on Flickr.

However, they also do quite a lot more.

HTML parsing, inlining and compression

O2 takes HTML files, inlines some of the included JavaScript and CSS files and removes some comments and carriage returns:

No escaping of > and < symbols takes place when the code is embedded. In some circumstances, depending on the DOCTYPE specified, this can generate errors and effectively render the web site useless.
If you are using a third party JavaScript code library which requires you to maintain their copyright message on their source code, or you simply want your own copyright message to always appear in your source code, you might be a little concerned to discover that O2 removes the copyright message as it passes through their network. Not sure what the legal ramifications of this are!
Inlining appears to be carried out in a non-deterministic way (possibly based on load times at O2's proxy server). Consequently, errors can be generated that appear to the customer to be occurring randomly.

On top of this the processes carried out by O2 will be increasing the latency of delivering the page to the user and completely messing up any optimisations put in place by the web site designer. If, for example, you place all the JavaScript at the end so that the page can be rendered on screen before everything loads, you are now having to wait, not just for everything to load, but also for O2 to assemble it all into one inlined file.

Where this micro-optimisation really starts to hit is where the user visits multiple pages of one web site which all load in common libraries like jQuery or Google Analytics. Instead of your local mobile browser caching these resources, O2 has embedded them, so they have to be embedded uniquely into every web page and sent to the user in their entirety. It's called a micro-optimisation because it optimises in the small individual case but in the bigger picture it actually slows things down massively and increases your bandwidth usage. The usage that you may well be paying for.

What are you paying for?

So, if you are an O2 customer, how can it be that O2 can modify the web content without telling you? How can they sell a service and secretly make changes to what you get? It may be well intentioned, but it is nevertheless a completely hidden modification that even O2 customer support are unaware of.

How can any web site designer try to support their web site if customers are actually receiving something different that O2 have modified on the way in a way that they don't tell you about?

Legally, can O2 strip copyright messages from code and merge code which may, by the nature of their licenses be required to be distributed separately with copyright messages left intact.

If you are an O2 customer or a web designer, I suggest you contact O2 and make them aware that this is a serious issue.

Lift: Sending mail through Google Mail

2009-11-26T10:50:00.006Z

I was trying to get my Lift web application to send out some mail through a Gmail account. I'm rusty with Java Mail so this took longer than it should have, so here's the code for anyone else it helps. The first big of code is just a bit of code to insert into the Boot class. I've put it in a separate method just to keep this code together, then I call it straight from the boot method.

import javax.mail.{PasswordAuthentication, Authenticator}

  private def setupMailerForGmail() {
    // Set up mailing to mail through Gmail account
    System.setProperty("mail.smtp.starttls.enable", "true")
    System.setProperty("mail.smtp.ssl.enable", "true")
    System.setProperty("mail.smtp.host", "smtp.gmail.com")
    System.setProperty("mail.smtp.auth", "true")

    // Turn on display of emailing processing logs by setting this property...
    //System.setProperty("mail.debug", "true")

    Mailer.authenticator = Full(new Authenticator {
      override def getPasswordAuthentication =
        new PasswordAuthentication("accountname…@gmail.com", "password…")
    })

  }

To send an email you can use something as short as:

import net.liftweb.util.Mailer.{To, From, PlainMailBodyType, Subject}
import net.liftweb.util.Mailer
…
    Mailer.sendMail(From("accountname…@gmail.com"), Subject("Just a test"),
      List(PlainMailBodyType("The body of the test message"), To("recipient_email_address…")) : _*)

Once this is working it's relatively straightforward to work out how to do XHTML mails and send to multiple recipients, etc.

The one bit of code that helped me the most was setting the mail.debug system property which caused the Java Mail code to output a trace of the mail sending process and allowed me to spot that I had failed to enable SSL.

Pragmatic Scala: Parsing columns from plain text

2009-11-26T09:49:00.007Z

A few days ago I was preparing for a meeting and I thought I'd print out a list of all the attendees from the Eventbrite event details.

When I looked at the list I saw that it seemed to be in no particular order and didn't print very well so I thought, "I wonder if I could copy and paste this into a spreadsheet and sort it by surname". Unfortunately each entry had one or two lines and variable information in it and was delimited by spaces and commas.

Then I thought, hmm, I could process this quickly with a bit of Scala and pattern matching.

There's nothing clever here, but for me it's quite significant that the effort involved in writing this code was so low that I was able to use Scala to solve this problem there and then and it worked seamlessly. I've posted it here because there are too few simple examples of Scala lying around the web for people to get started with.

So here's the code if you're interested…

#!/bin/sh
exec scala "$0" "$@"
!#

import java.io.File
import scala.io.Source

val source = Source.fromFile(new File("Attendees.txt"))
val lines = source.getLines.trim
lines.foreach( line => {
val (details, presentation) = line.split("\t").toList match {
  case List(d) => (d,"")
  case List(d,p) => (d,p)
}
val (name, position, organisation) = details.split(", ").toList match {
  case List(n) => (n,"","")
  case List(n,c) => (n,"",c)
  case List(n,p,c) => (n,p,c)
  case List(n,p,d,c) => (n,p+ ", "+d,c)
}
val (first, middle, last) = name.split(" ").toList match {
  case List(f) => (f,"","")
  case List(f,l) => (f,"",l)
  case List(f,m,l) => (f,m,l)
}

println(List(first,middle,last,position,organisation,presentation).mkString("\t"))
})

It was parsing lines of text which looked roughly like this:

Firstname [Middlename] Surname, [[Position, [Department ,]]Company][\t Presentation]

(e.g. bits in square brackets were optional).

For those unfamiliar with Scala, the first three lines just allow the code to be run as a script on the command line just like any shell script. Then the lines.foreach( line => code just loops through each line of code setting line equal to that line. The nice bit is the matching parts which assign values to multiple values at once so:

 val (details, presentation) = line.split("\t").toList match {
  case List(d) => (d,"")
  case List(d,p) => (d,p)
}

effectively says:

Assign to the variables: details and presentation at once, the value of the following expression.
Take the line and convert it to an array of Strings by separating it at every tab character, then convert the array to a List.
Match the List to one of two patterns:
If it matches the pattern of a List with one entry then assign the value of that entry to d and return the thing on the right hand side of the =>. In this case it is a tuple (pair of values): d and an empty String.
If it matches the pattern of a List with two entries then assign d to the first and p to the second and then return them both as a tuple.

So, at the end we have details set to everything before the tab and presentation set to everything after the tab or an empty string if there was no tab.

However, if there are two or more tabs then none of the patterns match so the program creates a runtime error which is great because we know that the program is either doing what it was meant to or not.

An end to restart problems on Mac OS 10.5.7

2009-05-13T12:44:00.002+01:00

I've been running on 10.5.7 for a little while before the official release and having problems with the machine refusing to shutdown properly. Yesterday I ran across a discussion board support comment that recommended removing items from your home directory ~/Library/Plug-ins directory and removing the top level /Library/Preferences/com.apple.loginwindow.plist file.

I don't know which one solved the problem because I couldn't be bothered doing one at a time, but I ended up removing an old DiskImages directory in my plug-ins which had a VirtualPCDiskImagePlugin.bundle in it that seemed to be creating errors in my log at startup, and I removed the loginwindow.plist file. Now everything seems to be working smoothly again!

This post marks a return to blogger, because I've decide to blog again from time to time and it's the easiest option for me right now!

This blog has moved

2005-03-14T16:39:00.000Z

I have now moved this blog to a Moveable Type based blog providing categories and TrackBack facilities.

The new URL for the blog is www.typingahead.com/management/.

The syndication feed has also changed and can be obtained either as RSS or Atom.

Please update any bookmarks or RSS subscriptions to see new posts. At some point in the future this site will be taken down.

PRINCE2 Open Exam booking workaround

2005-02-25T12:09:00.000Z

Attempting to formalise my PRINCE2 knowledge has been frustrated by the fact that the Open Examinations in the UK conducted by APM Group are booked through a web site booking form which which uses some outdated JavaScript to display available booking slots. From my testing this appears to only work on Internet Explorer on a PC, though Internet Explorer on the Mac will display the options even if you can't book them!

I'll see if I can persuade them to fix this, but in the meantime, if you are a user of Firefox or Safari you can circumvent their JavaScript museum by typing the following line (as one long line) into the location bar and hitting Return:

javascript:var str=document.f.T1.value; document.getElementById("Topic").innerHTML=str; document.getElementById("Topic").style.visibility = "visible"; alert("Done!");

Replace "T1" (for Milton Keynes) in the above with "T2" for Winsford or "T3" for York.

Once the dates are displayed you can proceed as normal.

Getting Things Done

2005-02-23T17:14:00.000Z

Perhaps I live in the dark ages, but somewhere down the line I’ve missed the “Getting Things Done” (GTD) movement and it appears with recent reading that I’m missing out, so I feel overwhelmingly obliged to read “Getting Things Done: The Art of Stress-free Productivity” by David Allen.

The final straw that lead me to this conclusion was reading one of today’s “Cutting Through” entries, “Prince 2, product and Getting Things Done”, which was about the 10th reference I’ve read to GTD in the last month.

An incomplete thought on blogging

2005-02-22T21:22:00.000Z

What is the point of a blog?

It’s an interesting question to which I can personally think of too many answers to consider writing about, but a few short ideas have permeated recently which seem worth noting and provide an amusingly recursive entry, if you get what I mean…

Blogs are about sharing useful information with like minded people

I read blogs because they inform me, so blogs are often about informing people. How do they inform? They inform because they share knowledge, and often they share a learning process. The writer moves from point A to point B in their understanding of a subject area, and the reader, who reads the blog because they feel an affinity for the writer, is also given a stepping stone to move from A to B without having to do all the work involved in discovering the step.

So, whilst a blog can be anything, in this understanding, a blog is about sharing what you are learning rather than attempting to write some creative or editorial masterpiece.

This being the case, a blog of this kind is not something that should take a long time to compose, because it doesn’t need too much thought, it just requires the thoughts to be composed on the screen and it doesn’t need to take long for another reason…

Blogs are about progressing knowledge not completing it

Normally writing is constrained by the need to complete things… the need to make something into a whole with a beginning, a middle and an end. From the perspective of a bit of prose this is probably still true if you want your blog to read well, but it doesn’t matter with regard to the thoughts it communicates.

Forget etiquette, forget the norms of old, blogging is about a conversation separated from time. I blog something and someone blogs a response somewhere else in the world at some time later. Perhaps I never hear the response, but someone else does, and it takes thoughts forward and builds on them.

What’s the point of completing a thought and wrapping it up in a parcel. For one thing it hits the old 80/20 rule that applied here says that wrapping up a thought takes 80% of the time and probably contributes 20% or less of the substance. It’s perhaps hard to admit that a thought isn’t complete and that we don’t have the final answer, but I know as a reader of blogs that I would be sorely disappointed if all the blogs I read had censored by the ‘completed thought’ police.

Funnily enough a blog could be described as agile publishing. Perhaps I should register that definition. Anyway, I could go on but…

Mind mapping with FreeMind

2005-02-21T23:10:00.000Z

From time to time, when I have a problem to solve I like to get out pen and paper and start drawing out ideas and thoughts and connecting them together. Using some random variation on mind mapping and Goldratt’s “Current Reality Trees”, I try to make order out of some disparate thoughts and often find the whole processes extremely helpful.

It’s hard to tell whether it’s the process or the product that matters, perhaps it’s both. Anyway, pen and paper and getting hard to use nowadays so from time-to-time I try to find a more appropriate tool to aid the process.

In the last few days I’ve taken another look at the free open-source application FreeMind. It’s a Java application so it works across platforms, and it currently has a new version 8 pre-release tucked away which takes away a number of the serious issues with the current full-release and makes this a real appealing application.

It’s a simple tool for ‘mind mapping’, designed to allow large mind maps to be created and navigated with minimal fuss. It has a ‘fairly’ intuitive interface and shortcuts that make the building process sufficiently quick to be really useful. Some serious bonuses include the ability to export to a nice selection of formats including PDF, and the fact that the underlying file format is XML, so there are endless ways by which the resulting maps can be repurposed, or by which other things can be converted into mind maps.

I will try it out some more and see how it stands the test of time.

Multi-platform VNC based automated testing

2005-02-16T12:07:00.000Z

Following the automated software testing theme I've just run across Eggplant an automated testing package which uses VNC to remotely control and watch a computer system under test. This allows it to work across a range of platforms. It can even record movies of the test taking place for later review.

For those unfamiliar with VNC, it is a cross platform solution to controlling and viewing the screen of another computer across a network connection.

There are two issues. Firstly, they don't give any pricing away on their web site, so it could be ridiculously expensive, but that said they do claim it is "affordable", so there's a basis on which to beat them down! The other issue is that it only runs from a Mac OS X based system, so development teams without Macs may have to fork out an additional £500 to buy a testing machine. Given the price of some other testing packages, this may not matter much, and at least it won't required daily patching to render it virus free!

Automated Acceptance Testing

2005-02-08T10:14:00.000Z

Yesterday I managed to get a long to the Agile Scotland meeting, here in Edinburgh which was looking at automated acceptance testing, and in particular, FIT and Extractor.

Although the talk was interesting, I wasn't entirely convinced by some of the points made.

The idea of automating acceptance testing seems extremely valuable. Acceptance testing is a chore and in reality the customer often fails to do it or to do it effectively. Indeed, in the worst cases the developer relies on the customer failing to do the acceptance testing properly! So, providing an automated tool for doing this testing allows the customer to fulfil their role with the minimum of effort, and provides the developers with a tool to guarantee that their code is up to the test, whenever they want to check it.

The distinction between unit testing and acceptance testing is a little ambiguous. Unit testing is distinguished in that it should be testing smaller units very quickly so that it can be performed as part of every compilation/build cycle. Acceptance testing should be testing the entire package (including the interface) to ensure that everything works together in the way that the customer expects. Acceptance tests do not have to be carried out all the time so they can take longer to perform. In reality a quick test that exercises the whole application would be a useful unit test, and automated acceptance testing is not always easy to do, so it is sometimes necessary to exercise units in isolation: perhaps bypassing the interface, or directly injecting data to provide as test input.

So one of the main advantages of FIT and Exactor is that they present the tests in a form that is more user-readable than your average bit of unit testing code. This allows customers to see and understand the tests as well as seeing and understanding the test results. In theory it also allows customers to write tests, but it was acknowledged that this rarely happens in practice.

Despite some of the ways in which the information was presented, it would appear that these acceptance testing tools are basically layered on top of jUnit, with Exactor being an abstraction of FIT to provide plain text entry of tests rather than using HTML tables as FIT requires.

My main concern is that this solution fails to acknowledge reality. If customers are not really going to write tests themselves, but may do so with the help and assistance of professional testers or developers, then it makes more sense to provide tools which automatically present automated tests and their results in a user friendly way, rather than asking programmers to use yet another entry syntax to code the tests. Having said this, I can see genuine value in the way that these tools enable customers to subtly amend existing tests directly.

Ultimately I think that Apple's scripting technologies provide a nice analogy that is worth looking into.

For many years Apple has provided end users with a scripting language called AppleScript which was designed to allow users (not programmers) to script and automated repeated or complex tasks. The language had a very plain English feel to it that made it easy to understand existing scripts, but gave the illusion that you could write scripts by writing plain English when in reality it was just as demanding of syntax and structure and was very bad at explaining your coding errors when you went wrong.

Apple is just about to release a new version of their operating system (Mac OS 10.4, code-named 'Tiger') which contains a new application called "Automator". This provides a GUI for building scripts which allows you to drag and drop linked operations in such a way that the interface will only allow you to link compatible operations and will only allow you to link them in a suitable order. It doesn't stop you doing pointless or stupid things, but it ensures that you produce a 'script' which works.

Automated Acceptance Testing involving the customer directly will really come into its own when the developer community spends the considerable time involved in making a friendly user interface that will hand-hold customers through the process and allow them to create and edit tests without having to understand any kind of syntax at all.

I almost forgot to mention that there was a reference made to Selenium - a tool for automated testing of web applications directly through a range of browsers. I haven't had time to look into how the tests are defined, but it looks very useful in principle.

UK almost bans spam messages

2003-09-22T09:38:00.000+01:00

I notice that the BBC are now reporting on a new law to ban spam messages. But as the article itself says, the law will not make it illegal to send spam to business email addresses. So that's not exactly going to do a lot for the efficiency of British industry.

Using Palm Photo on a Mac with the Tungsten T2

2003-09-12T13:52:00.000+01:00

I spent a good deal of time trying to find an application to display photos on my Tungsten T2, puzzling over why the documentation refers to a Photo application, but it doesn't seem to be on the installer disks. Then I noticed the Add-on folder in my Palm Desktop installation and discovered a whole set of .prc files which turn out to be the very thing. I don't know if I did something wrong with my initial installation or what, but I can now view my photos on my palm without forking out for another application.

Project Management on Palm

2003-09-11T12:41:00.000+01:00

Just come across an open source project management package Progect Manager for Palm. So far it looks very promising though it appears to have a fault in the 'flat view' which misses out parent to do items.