Daily Telegraph 12th August 2014

Internet dongles plug into a computer and allow users to wirelessly connect to the internet

You may have noticed yesterday that your internet connection was rather sluggish, or perhaps went down entirely. You were not alone: problems were reported around the world.

Auction site eBay, for instance didnt work.  The company has not explained the exact nature of the problem, but admitted in a statement that “technical experts identified this was due to upstream Internet Service Provider (ISP) issues”. Password manager LastPass was also affected, leaving customers locked out of their accounts.

The issue, according to many experts, was with something called the Border Gateway Protocol (BGP). You may never have heard of it, but it is absolutely vital to the operation of the internet and is causing large problems.

BGP is what tier-one ISPs, your last-mile ISP and various large networks use to route data from their own machines to others, and vice versa. When you visit a website, that data bounces all over the world, through machines belonging to all manner of companies and organisations. To make this work, machines called routers (large commercial versions of what you have at home) keep a table of known, trusted routes through the tangled web.

This routing table has been constantly growing in size as the internet expands and becomes more complex – more information needs to be stored in order to allow the router to bounce data to the correct destination along a logical route. Until late 2001, the size of the table was growing exponentially, which was clearly unsustainable. A big effort to implement more efficient methods was made which temporarily slowed expansion. But it didn’t last long.

Now we are at the point where some older routers are struggling to cope: their memory is too small and their processors not powerful enough. A full copy of the routing table now contains 512 rows of 1,000 ports, a total of 512,000 routes. Older hardware was never designed with larger tables than that in mind. Many have a strict 512,000 route limit, put in place by programmers many years ago who were forced to arbitrarily choose a number; you don’t make something so capable that it can operate for a hundred years as the hardware cost would be enormous, but you must also ensure a practical lifespan. The result if often little more than an educated guess.

As these machines struggle, the effect is that ISPs experience outages, hosting companies have problems and websites either go down or slow down.

The problem has been anticipated for years, but replacing these machines is an expensive, non-trivial task. It’ll happen – especially now that BGP issues are causing such large problems – but not overnight.

Many are already back up and running, others were replaced months or years ago. More problems will also be alleviated slightly by a new protocol called IPv6 which will reduce the strain on BGP. But further outages cannot be ruled out.

Web hosting company Liquid Web tweeted yesterday that: “As ISPs have recovered from #512k active BGP routes being reached, many of our customers affected by these carrier issues have regained the ability to reach their sites. We are still currently up, working to get a timeframe when sites can be reached from all locations and from any ISP.”