"show your working"—a plea for Scala coding quality

The recent debate about the complexity of Scala over Java and other languages has lead me to bemoan the quality of some significant chunks of Scala code currently circulating in the public domain.

I probably should just be patient as the cycle of programming language maturity is a common one. In the early days of a language, in the absence of coding standards and common practice, people grab at the new features of a language and enthusiastically use them with little awareness of the consequences. As a language matures, the community of developers adopts and polices standards to the extent that a newby developer soon finds themselves instructed on the basics of what to do or not to do.

There's a lot of Scala code out there that feels like it's been put together by spotty adolescents with too many hormones and not enough common sense. Okay, I'm being harsh, I'm a Scala newby too, but I think it really is time we started to get a bit more focused on developing some good coding practice and mentoring each other towards maturity.

I'm going to start with a simple issue which is close to the recent complexity debate...

Extremes of pure functional style encourage code obscurity

Functional programming is great, but taken to an extreme functional programming can:

• encourage short meaningless function names.
• discourage temporary named variables which help to explain what is going on.
• encourage the creation of large numbers of small functions which obscure the modular interfaces.

Take this function as a slightly extreme example:

  def ^!!||^(implicit f: Foldable[IN]) :
      Kleisli[Option, String, NonEmptyList[List[Char]]] =
          kleisli((p : String) => {(this !!|| p).toNel })

I'm not trying to pick on anyone here, so apologies to the author, but this method is one of over 50 public methods in a single Scala trait of an open source Scala library. Is it any surprise that people think Scala is complicated?

So, here are my Scala programming tips for the day:

1. Minimise your modular interface: in other words, keep your publicly accessible functions, properties and classes, etc. to a minimum. If it doesn't need to be public make it private. Why? ... because developers reading your code shouldn't be left swimming around trying to find what's important and what isn't. If you create a function that is only used by one other function, define it within the function that requires it. This makes it absolutely clear that it is only there to support the other.
2. Use meaningful names and avoid symbols: if your readers can't figure out what a method is supposed to do then chances are neither will you in a few months time! The use of symbols for function names is a curse of the current trend in DSLs (Domain Specific Languages). Yes, do define a + method to add two objects together, but don't write functions called, ~>, ~~> and ~%> and think you're clever!
3. Don't be afraid to use vals to store partial results in your function rather than trying to string everything together into one long expression. Not just does this simplify the code by breaking it down into smaller parts, it also provides you with the opportunity to "show your working" by assigning partial results to values with meaningful names. I doubt an assignment to a val has any impact on performance. But I'll leave it someone more qualified in bytecode to back me up on that.

Okay. That's me done for today. Don't miss out on this great Scala community style guide. A really valuable contribution to Scala maturity.

The official way to bypass data modification on O2 mobile networks

Last week I received details from O2 of the official way of bypassing the 'optimisation platform' that O2 use on their mobile networks. They were particularly concerned that I pass on the following comment on the use of this and so I include it below verbatim:

when using this “bypass” function you must consider page impressions will take longer, on average, and thereby detract from the user experience, a slower experience. Also, a greater volume of data will be downloaded , on average, and those customers who do not have unlimited data will use their data bundle faster or incur high bills.

They then then went on to reference section 14.9.5 of the W3C HTTP 1.1 Protocol specification: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.5 which refers to the use of the HTTP Header "Cache-Control: no-transform" response directive. By setting up your server to return this response header, O2 indicate that they will not modify the data.

I should add that I haven't had a chance to test this yet and in particular, to test whether this stops the compression of images as well as the modification of HTTP source code.

Whilst this appears to offer a way for web developers to prevent their site content from being modified, it does not resolve issues for developers of web applications which utilise web data feeds that are not under their own control. For example, the developer of the great iPad Viewfinder app which provides Flickr photo search and download cannot prevent O2 reducing the quality of the images when using it over their mobile network.

O2's response to data modification queries

Just over a week ago I had a conference call with Tim Fielden (Data Access) and Gavin Sheldon (Head of Networks O2 UK).

1. They indicated that they had looked into the legal issues I had raised and were advised that the data modifications they are making do not cause web site developers to infringe any copyright or licensing terms. As I said to them at the time, I still find it hard to believe that O2 themselves can modify other peoples data without informing them.
2. They recognised a need for developers to be able to bypass their data modifications in more exceptional circumstances and said that they would send details of how to do this. I'll explain this next in a separate entry so it's easier to search for in future.
3. They expressed confidence in their extensive testing of their network and data compression/modification systems and said that, in their testing, they found that these network optimisations resulted in the best customer experience. I specifically queried whether or not they tested accessing multiple pages in a row to which they reiterated their confidence in the extensiveness of their testing, but welcomed any specific cased of web sites that were found to be underperforming on their network.

All in all they were very polite and clearly expressed their desire to be helpful but I was left feeling that it was going take more than a few customer complaints before they would do anything to stop modifying data without telling anyone. No-doubt they feel that that, as not the only operator to be doing this in Europe, they don't need to rush to change the status quo.

Operators modifying mobile data content (a table)

Following my request for people to test their own mobile provider, here is an up-to-date summary of responses so far. The number under "Modifies" or "Doesn't appear to" is the number of individual confirmations I've received. I won't update this after I've received 4 consistent responses.

Please note that "modifies" means that the simple javascript test appears to show that the provider is inlining CSS style sheets. The "Doesn't appear to" means that the test didn't detect CSS inlining, but doesn't exclude the possibility that the operator is still modifying content in a different way.

Country	Operator	Modifies	Doesn't appear to
Australia	Optus	-	1
Canada	Fido	-	1
Germany	Vodafone	2	-
Netherlands	T-Mobile	*	* (Note 1)
UK	O2	4	-
UK	Orange	-	1
UK	T-Mobile	2	-
UK	Three	-	2
UK	Vodafone	-	1
USA	AT&T	-	1
USA	Speakeasy	-	1

1. T-Mobile in the Netherlands apparently gives users a choice when they subscribe to the service.

O2 Data Modification Update

A number of people have asked for an update on the O2 data modification situation. Here is a quick list in no particular order:

1. The Head of O2 Networks has responded to my communications and has arranged a meeting later this week to discuss the issues. He has also asked the O2 legal team to reassess the licensing implications of the modification of licensed source code including the removal of copyright messages.
2. The data modification technology used by O2 appears to be provided by the company ByteMobile which appears to suggest that some of its technologies are also in use by Orange, Vodafone and T-Mobile in the UK.
3. A 'back of an envelope' calculation appears to indicate that the data modification taking place will indeed significantly increase the amount of data being sent to your mobile device when you access two or more pages of a standard web site like the BBC news or Guardian news web sites. This is on the assumption that your mobile device would normally locally cache CSS and JavaScript files. It's quite possible that some older mobile devices do not do this.
4. The Register (online technology publication) has responded to me about this issue but to date I don't believe that this issue has been published in any press article. I think people generally find it a little too technical to write about. Everyone I have spoken to has been completely unaware that this practice is going on.
5. A comment making reference to this issue that I added to a BBC news article was removed as being "potentially defamatory".
6. Following issues arising with another web site, this blog has followed up on the same issue: O2 UK mobile users - your operator is breaking this site for you..
7. StartupCafe kindly posted an article on my investigations: O2 Compression on mobile devices.
8. I have contacted Ofcom who are assessing my complaint.

Is your mobile provider modifying your web pages? Instant test.

Prompted by an email earlier today I've produced a very simple web page that carries out a very simple test that checks whether one of the modifications to web pages has been carried out when the page is loaded.

If you access the page Inlining Test on your mobile device whilst connected through your mobile network (not through WiFi), you should see one of two messages:

1. "No identified modification has happened to this page"; or
2. "This page has been altered! It appears that the stylesheet on this page has been inlined by your network provider."

It would be really helpful to get some feedback on which providers, mobile devices and APNs are exhibiting issues and in which parts of the world. If you could leave a comment if you have a problem on a setup that hasn't already been listed then that would be great.

Please note that this test hasn't been extensively tested so I can't guarantee it will work on every mobile browser. Feel free to let me know if you spot a gaff!

For those interested to know the page contains a simple JavaScript that checks to see that the stylesheet link on the page is still there once the page has completed loading. If it isn't there (most likely because of the Bytemobile filter inlining the JavaScript) it displays the message to say that. This page also has Google analytics on it so that I can get a feel for how many times people are using it.

Letter to Ronan Dunne, CEO of O2

Dated 23 July 2010:

Thank you for your response.

The devices I have personally used to confirm that data modification is taking place are an iPhone and iPad. However, I have received confirmation from other users that that modifications are taking place for mobile broadband users using laptops as well, and I know that many of the practices are also occurring for users with other mobile phone devices.

In specific reference to the UK Government web site. The UK government web site uses the SWFObject JavaScript library which is made available under an MIT license which requires that the license is distributed with the software. It uses the jQuery JavaScript library which is also distributed under a joint MIT and GPL Version 2 license. I believe the GPL license prohibits the software being combined with any other software or modified. There are other works also used by the UK Government web site.

Currently, as far as I can tell, the O2 mobile network is running a caching proxy server which is combining JavaScript libraries into the main HTML source of web pages and removing copyright messages. I believe, and the informal advice I have received would confirm that this action is in contravention of the licensing terms. As it is O2 that are doing this then I believe that O2 is breaking the law. I should also make clear that I referred to the UK Government web site to try to ensure that my email was not ignored. These practices are being applied to every web site as far as I can tell, so there will be literally millions of other examples.

However, the broader and arguably more serious issue is the fact that O2 has a practice of modifying any data passing through their network between senders and receivers without notifying either party of the changes that are being made. Even when I tried to find out what was happening I was told by your technical department that they *did* compress data but that they were not legally required to give any detail of what they did. (I should add that this statement was misleading as they do a lot more than just compress data, they also change and remove data). In other words, even if I ask, I am not told what O2 are doing to my data.

I believe that this practice may nullify the contractual obligations of every customer of O2's mobile data services at the present time.

Finally, I refer you to European legislation being formed around 'net neutrality' <http://europa.eu/rapid/pressReleasesAction.do?reference=SPEECH/10/153> and curently being considered by Ofcom. The Vice President of the European Commission said: "Transparency is non-negotiable. This is already addressed in the new regulatory framework, but the principle is worth re-stating: in a complex system like the internet, it must be crystal clear what the practices of operators controlling the network mean for all users, including consumers."

Kind regards,

Stuart Roebuck.

Correspondence with O2 about their data compression policy

Here is O2's response (22 July 2010 emphasis mine) to my enquiry asking for an explanation of their policy regarding modifying data sent over their mobile network. For those readers who are not clear on the technical details, the blanket statement that any phone compresses the data for better viewing is not true.

Hello Stuart

Thanks for your email about the data compression on your iPhone.

I get to know from your email that you wish to know our data compression
policy.

Just to let you know Stuart, that if you are accessing the internet on 
any phone, the phone automatically compresses the data to allow better 
viewing of the websites.

This is a technical query and we've a separate team which is specially 
trained on all the technical issues. So that you can get a satisfactory 
response to your query, all you need to do is speak to this technical 
team on 08448 752 302. We're open: 

- Monday to Friday between 8am and 9pm. 
- Saturday between 8am to 8pm
- Sunday 8am to 6pm.

Calls to the above number are free from all O2 Pay Monthly mobile 
phones. You can also dial 2302 free from an iPhone to speak to our 
iPhone customer service team.

I wish you a great day ahead. 

Kind regards

------
O2 Customer Service

Telefónica O2 UK Limited, Registered in England No 1743099. Registered 
Office: 260 Bath Road, Slough, Berkshire SL1 4DX.

I followed up on the number they gave but reached another customer support staff member who did not know what a "data compression policy" was and cut me off when they said they'd pass me on to someone who might.

I tried again and this time the person told me that I hadn't got through to the "technical team" and that the number I had been given was "just a generic number". He said he didn't know anything about the issue and nor did his manager but he would try and put me through to a different department that might.

He then came back again and told me that he could only pass me any further information if I confirmed that I was an O2 customer and gave my account details.

Finally, after approx. 2 hours on the phone I was told that O2 could official confirm that they "compress data" and they were not legally required to give any further information about what they do.

I'm now awaiting a response from O2's CEO.

Compiled list of web references to mobile compression - mostly O2

O2 (UK)

• O2 discussion forum: "Disabling image compression on PAYG"
• Apple discussion board: "Is O2 using a proxy and compressing images?"
• O2 discussion forum: "iPad unable to bypass image compression"
• Apple discussion board: "Can I improve the image quality in Safari?"
• Hello! I'm Ash Matadeen blog: "Get rid of image compression on O2's network "
• kcjh's world blog: "Removing O2's Image Compression on Mobile Web - Handy for iPhone users!"
• EdAndersen.com blog: "iPhone O2 – how to fix the image compression"
• The Mobile Broadband User blog: "O2 mobile broadband compression continued"
• The Mobile Broadband User blog: "Fix for image compression on O2 mobile broadband"
• The Mobile Broadband User blog: "O2 delivers Internet via mobile broadband – but what Internet?"
• ByteMobile: O2 UK Enhances Native Browsing Capabilities of Apple iPhone with Bytemobile Web Optimization
• What's up on APRS.FI blog: O2 UK mobile users - your operator is breaking this site for you.

Note that all references to being able to use the "APN" username "bypass" on the O2 network to bypass image compression no longer apply for iPhone users. O2 appears to perform image compression and data modification regardless of this setting.

Vodafone (UK)

• 3G Mobile Phone Discussion forum (UK Only): "Image compression?"

Optus (Australia)

• ZDNet: "Optus' blurry iPhone pics are unacceptable"

Other

• Ofcom: Discussion document on: Traffic Management and ‘net neutrality’
• Vice President of the European Commission Commissioner for the Digital Agenda: Address at the ARCEP Conference (L'Autorité de Régulation des Communications Electroniques et des Postes) on net neutrality
• Vision mobile blog: Boosting internet in mobile: the return of the browser proxies (mobile megatrend series)
• Sourceforge: Rules for Responsible Reformatting: A Developer Manifesto

Oh and my own...

• Stuart Roebuck blog: "Mobile Proxy Cache content modification by O2"
• Stuart Roebuck blog: "Should mobile operators be free to modify content they deliver?"
• stackoverflow: Web site exhibits JavaScript error on iPad / iPhone under 3G but not under WiFi
• Stuart Roebuck blog: Letter to Ronan Dunne, CEO of O2

Should mobile operators be free to modify content they deliver?

My recent problems with delivering a web service to customers of the O2 network in the UK has got me thinking a lot about what we expect from mobile network providers.

I was trying to think of an analogy with something we are more used to and realised that web content delivery is very much like parcel delivery: we ask for something to be sent to us, they put it in a parcel and a delivery company are responsible for delivering it to us as quickly as possible without damaging it on route. In the internet context, we ask for a web page and we rely on the internet to deliver the page to us without modification.

However, it appears that we have fairly silently allowed the mobile networks to change this arrangement. For some time now we have been, wittingly or unwittingly, allowing many of the mobile networks (O2, T-Mobile, Virgin, Vodafone to name a few in the UK) to reduce the quality of images that we download. This seems fairly harmless and well intended. If it enables us to surf the internet quicker on our mobile phones when we can't see the images well anyway, then surely this is a good thing?

But it's not as simple as that.

What if I want to send someone an image over a mobile network and the image quality really matters. Firstly, I probably won't realise that O2 are going to modify it on route and may discover too late that the lower quality image has gone to print. Secondly, even if I do realise, how do I make sure that the important image isn't compressed? There is no option to bypass this. There is no way of getting hold of the content we have requested. O2 have substituted what we asked for with something inferior and they do it on purpose without asking us or giving us a choice to opt out.

It really calls into question the whole industry of web services. If you pay a web provider for some data and your mobile operator substitutes something different as it passes through their network, who has failed to deliver? If you pay for Flickr's Pro service and use it on your iPad, can you complain to them if the images you view are substandard? Will Flickr have any idea that the images they are sending you are not the ones you get?

And it's not just about images and image quality.

O2 may be going further than other providers, I really don't know. But I do know that they are modifying the very source code of the web pages you access over the internet. So the web services that painstakingly produced and tested by companies around the world are being modified by O2 in such a way that they may not work when they reach your mobile device or 3G connected laptop. You might blame your mobile phone or iPad whilst totally unaware that the problem is actually because O2 have changed the content enough to break it.

Can web content providers test all their services through every mobile network in the world to make sure that you get the service you have paid for? Of course not!

Should you, as a customer, have to figure out whether the problems you are having come from your mobile provider or the original content provider? Of course not!

Mobile phone networks should be reliable and they should provide the content you ask for without modification. If they want to optimise their networks then they can give you the option of accepting a modified network, but it should always be possible to turn off the modifications quickly and easily if anything goes wrong.

The craziest thing about this story is that, as far as I can work it out, O2's optimisations are probably increasing the load on their network and increasing the latency of their network. I have a suspicion that if they turned it all off they would save themselves some money overnight!

I have engineered a partial workaround for the web site I host and it now works on the O2 network and works quicker than it did before!

Mobile Proxy Cache content modification by O2

If you connect to the web on a mobile Safari device such as an iPhone or iPad using 3G over O2's network in the UK, then you may not realise that what you are seeing has been modified by O2 between the web site and your phone. If you are the author of a web site and find that some customers are not seeing what they should you may be frustrated to discover that O2 have changed the source code before it gets to your customer and that you have little way of finding out what they have done.

I'm sure O2 is doing this in order to speed up the service and reduce network load, but unfortunately their modifications are not without consequences, as I discovered trying to host a site designed specifically to work well on an iPad.

Image reduction

The most widely recognised change they make is to reduce the resolution of image files.  This has caused some complaint recently from iPad owners who have a large enough display to easily spot the different in quality that results when, for example, browsing photographs on Flickr.

However, they also do quite a lot more.

HTML parsing, inlining and compression

O2 takes HTML files, inlines some of the included JavaScript and CSS files and removes some comments and carriage returns:

• No escaping of > and < symbols takes place when the code is embedded. In some circumstances, depending on the DOCTYPE specified, this can generate errors and effectively render the web site useless.
• If you are using a third party JavaScript code library which requires you to maintain their copyright message on their source code, or you simply want your own copyright message to always appear in your source code, you might be a little concerned to discover that O2 removes the copyright message as it passes through their network. Not sure what the legal ramifications of this are!
• Inlining appears to be carried out in a non-deterministic way (possibly based on load times at O2's proxy server). Consequently, errors can be generated that appear to the customer to be occurring randomly.

On top of this the processes carried out by O2 will be increasing the latency of delivering the page to the user and completely messing up any optimisations put in place by the web site designer. If, for example, you place all the JavaScript at the end so that the page can be rendered on screen before everything loads, you are now having to wait, not just for everything to load, but also for O2 to assemble it all into one inlined file.

Where this micro-optimisation really starts to hit is where the user visits multiple pages of one web site which all load in common libraries like jQuery or Google Analytics. Instead of your local mobile browser caching these resources, O2 has embedded them, so they have to be embedded uniquely into every web page and sent to the user in their entirety. It's called a micro-optimisation because it optimises in the small individual case but in the bigger picture it actually slows things down massively and increases your bandwidth usage. The usage that you may well be paying for.

What are you paying for?

So, if you are an O2 customer, how can it be that O2 can modify the web content without telling you? How can they sell a service and secretly make changes to what you get? It may be well intentioned, but it is nevertheless a completely hidden modification that even O2 customer support are unaware of.

How can any web site designer try to support their web site if customers are actually receiving something different that O2 have modified on the way in a way that they don't tell you about?

Legally, can O2 strip copyright messages from code and merge code which may, by the nature of their licenses be required to be distributed separately with copyright messages left intact.

If you are an O2 customer or a web designer, I suggest you contact O2 and make them aware that this is a serious issue.