Tuesday 20 July 2010

Mobile Proxy Cache content modification by O2

If you connect to the web on a mobile Safari device such as an iPhone or iPad using 3G over O2's network in the UK, then you may not realise that what you are seeing has been modified by O2 between the web site and your phone. If you are the author of a web site and find that some customers are not seeing what they should you may be frustrated to discover that O2 have changed the source code before it gets to your customer and that you have little way of finding out what they have done.

I'm sure O2 is doing this in order to speed up the service and reduce network load, but unfortunately their modifications are not without consequences, as I discovered trying to host a site designed specifically to work well on an iPad.

Image reduction

The most widely recognised change they make is to reduce the resolution of image files.  This has caused some complaint recently from iPad owners who have a large enough display to easily spot the different in quality that results when, for example, browsing photographs on Flickr.

However, they also do quite a lot more.

HTML parsing, inlining and compression

O2 takes HTML files, inlines some of the included JavaScript and CSS files and removes some comments and carriage returns:

  • No escaping of > and < symbols takes place when the code is embedded. In some circumstances, depending on the DOCTYPE specified, this can generate errors and effectively render the web site useless.
  • If you are using a third party JavaScript code library which requires you to maintain their copyright message on their source code, or you simply want your own copyright message to always appear in your source code, you might be a little concerned to discover that O2 removes the copyright message as it passes through their network. Not sure what the legal ramifications of this are!
  • Inlining appears to be carried out in a non-deterministic way (possibly based on load times at O2's proxy server). Consequently, errors can be generated that appear to the customer to be occurring randomly.

On top of this the processes carried out by O2 will be increasing the latency of delivering the page to the user and completely messing up any optimisations put in place by the web site designer. If, for example, you place all the JavaScript at the end so that the page can be rendered on screen before everything loads, you are now having to wait, not just for everything to load, but also for O2 to assemble it all into one inlined file.

Where this micro-optimisation really starts to hit is where the user visits multiple pages of one web site which all load in common libraries like jQuery or Google Analytics. Instead of your local mobile browser caching these resources, O2 has embedded them, so they have to be embedded uniquely into every web page and sent to the user in their entirety. It's called a micro-optimisation because it optimises in the small individual case but in the bigger picture it actually slows things down massively and increases your bandwidth usage. The usage that you may well be paying for.

What are you paying for?

So, if you are an O2 customer, how can it be that O2 can modify the web content without telling you? How can they sell a service and secretly make changes to what you get? It may be well intentioned, but it is nevertheless a completely hidden modification that even O2 customer support are unaware of.

How can any web site designer try to support their web site if customers are actually receiving something different that O2 have modified on the way in a way that they don't tell you about?

Legally, can O2 strip copyright messages from code and merge code which may, by the nature of their licenses be required to be distributed separately with copyright messages left intact.

If you are an O2 customer or a web designer, I suggest you contact O2 and make them aware that this is a serious issue.


Stuart Roebuck said...

Ash Matadeen helpfully pointed me to his blog entry about bypassing this for the iPhone. Unfortunately this doesn't work for iPad: http://www.ashmatadeen.com/get-rid-of-image-compression-on-o2s-network. In fact I've just tested it on the iPhone and I think it nolonger works there either.

Stuart Roebuck said...

Since writing this post I have discovered that O2 is using the a web compression technology produced by a company called Bytemobile. Some more details can be found here: http://www.bytemobile.com/company/customers/web-optimization.html

Stuart Roebuck said...

Here's a link to a another blog post from a company that has experienced the same problems with O2 breaking their web application. O2 UK mobile users - your operator is breaking this site for you.. Many thanks for their kind links to my blog entries.

Ash Matadeen said...

Interesting blog post, Steve. I hadn't realised that they also 'optimised' the source code. This should be a punishable offence!

Unfortunately, I'm no longer with O2, so can't test whether the compression bypass settings in my blog post still work. I can only suggest - if you haven't done that already - that you clear the cache in Safari before loading the page with the new settings.

I'm beginning to think that O2 is not the only network messing up our data without authorisation. It looks like the maps on http://aprs.fi/ doesn't load on the Vodafone 3G network either.

It's outrageous that the mobile phone companies do this; even more so that they do it without letting their customers know!

David Sim said...

Thanks for this post. I updated one of my iPhone apps to check for a comment in the returned HTML code. This endured the user was properly connected to the Internet, rather than being connected to public WiFi but not logged in.

It would fine on WiFI but not on O2 because the comment is stripped out. Easy to work around, but good to know.

Anonymous said...

Hi Iam Prabhu from chennai,joined today in this forum... :)

Anonymous said...

Hey, I am checking this blog using the phone and this appears to be kind of odd. Thought you'd wish to know. This is a great write-up nevertheless, did not mess that up.

- David

Anonymous said...

hi, new to the site, thanks.

Anonymous said...

Appreaciate for the work you have put into the article, this helps clear up some questions I had.

Anonymous said...

I have to confess that i sometimes get bored to read the whole thing however i think you'll be able to add some value. Bravo !

Kam said...

This drove me nuts all day! I tried document.write but it didn't work for me (same problem, works on wifi but not on 3g). I'm on T-Mobile UK tho.

Instead I renamed the files .bin instead of .js Thats seems to have done the trick :)

Make sure to put the type=text/javascript in tho of course!

Anonymous said...

what I was looking for, thanks

Paul Holmes said...

Further to the concern about copyright notices being stripped from code, when I lasted tested the O2 image compression behaviour, it stripped any EXIF data which is also a potential repository for copyright information.

O2 did not respond to my emails challenging them as to wether they had the right to materially alter material they do not own.

Unknown said...

I have had repeated problems browsing on two phones with an O2 configuration... even (especially) on Wi-Fi. Even when network is ok, proven by ping.
Removed O2 proxy setting, suddenly browsing is faster. Will see if it fixes Wi-Fi problem too...

Google Analytics Alternative