Tracing the Syrian Blackout
Updated Monday morning to include detailed Syrian network map, and include one-second BGP plots during the day of outage. --jim
Thanks to everyone who's stopped by this week to read about the Internet blackout that affected Syria last Friday. We're always glad to hear your comments, especially when you fill in some of the missing parts of the story that aren't obvious from the data.
A lot of the questions we've received about Syria this week have speculated about a repeat performance of last week's Internet blackout. Would the Syrian Internet get shut off again this Friday, as it was last Friday? Would it be visible in the BGP routing table? Would this become a recurring tactic, just another mechanism for crowd control?
Initial data from Google's transparency report suggest that total traffic was down, even compared to a normally quiet Friday.
But looking at the broader picture, a couple things are clear. First, there was no repeat of last week's event, in which two-thirds of all Syrian networks became flatly unreachable from around the world, an Egyptian-style disconnection at a very fundamental level. This week, while traffic levels were reduced (perhaps throttled or rate-limited, as in Iran), the routes themselves remained intact.
If you wanted to reach a Syrian website Friday, or if a Syrian browser wanted to reach a European website, the paths were known and the lines were open. Whether you could actually get enough bandwidth to upload videos, or make a Skype call, was another story. We've heard some anecdotal evidence that there were connectivity problems. But anecdotes aren't data.
Speaking of data, that's another question we get all the time. How do we examine a market like the Syrian Internet, and make our call about the structural availability of routes to its networks? Let me take a moment and describe some of the technology behind these reports.
Finding All The Paths
Many people assume that we somehow have direct access to the world's traffic data, measuring the rates of traffic on millions of network links as they ebb and flow with the rising and setting of the sun — a poetic vision, but obviously impossible (what a measurement challenge, to be everywhere all at once!)
Instead, our information comes from two indirect sources: changes in the BGP routing table (all the myriad ways that global ISPs believe they can reach Syria at any given moment) and active traceroute measurement.
First, BGP. In this plot, you can see the percentage of Internet transit (top) and total Internet transit of customer base (bottom) that Syrian Telecom, AS29386, sends to each of their international ISPs. This plot displays changes as small as one second in duration, meaning that we should see any differential fluctuations in Syria Telecom's transit spectrum, no matter how brief. In Egypt, which has much more complex provider structure, we saw a rocky ride to the bottom, before the outage was complete.
In fact, the Syrian transitions are clean and sharp, with a single outage, followed at 19:00 the next day by a partial restoration, followed by the remaining restoration Saturday morning. PCCW was relatively most affected, going to zero for the duration (no service to Syria). Deutsche Telecom, Turk Telekom, and Tata's relative standings are basically preserved.
Active Measurement of the Syrian Outage
Having looked at the routes, we can look at what active measurement has to say. For that, we use multipoint traceroute.
Traceroute is an old, old Internet protocol. You send small probe packets towards a destination, with different expiration times, and when they come back from intermediate points along the route, you see where they've been and how long they've been travelling. We do that 24 hours a day, from computers all over the world, building up a coherent picture of all the paths traffic is actually taking into networks in Chicago, or China, or Syria.
To show you what I mean, here's a picture that summarizes our traceroutes into the 60-odd networks that make up what we call "the Syrian Internet." Time flows from left to right, covering about 2 weeks. The vertical lines-and-boxes show you the latency distribution of packets going to Syria and back to one of our collectors worldwide.
The small red line is the median value, normally about 230 milliseconds for a packet to travel into Syria and back out again. When networks get really busy, and traffic exceeds the capacity of the Internet connection, they will sit in router buffers for a long time, and this latency measurement will edge upward, sometimes by a factor of 2 or 3 times more delay. Here, you don't really see that in evidence — there's some fluctuation, but no consistent upward surge in delay at any point over the two-week period.
In fact, median latencies drop during some of the June 3rd event, suggesting that the routers that remained reachable were better-connected than average: close to the Syrian backbone, fewer hops away. That would be consistent with the theory that access networks around the country (DSL, 3G mobile) were down, and government/core telecoms resources remained up.
Of course, last Friday's event is clearly visible. That's the second dataset you see plotted as a blue background: the total number of traces inbound to Syria that successfully completed. What happened during last Friday's day of rage was a real disconnection: the withdrawal of routes. Many of our traceroutes into Syria that day simply couldn't get started, because they had no idea how to find their way into Syrian Telecom. After the Syrian blackout was lifted, things basically returned to "normal," just as we saw after the (much longer) Egyptian blackout in January.
Looking through this particular lens, we see that Friday June 10th, looks nothing like Friday June 3rd. There's some reduction overall in the number of successful traceroute completions this week, compared to the same days last week. But there's nothing visible here to distinguish Friday 10th in particular as a day of rage. Both completion counts and measured latencies look the same Wednesday, Thursday, and Friday.
Another Perspective: Provider Choice
Okay, let's look through one more lens at the same event, showing how paths changed within the traceroute dataset. Remember, traceroutes don't just show you the delays for traffic into and out of Syria, they show you which providers' routers were encountered on the way in and out. That can tell us something about what those providers are up to, when there are big transit changes.
Again, this plot covers 2 weeks from left to right. The Friday June 3rd event is clearly visible as a drop in the total number of traces into Syria through all international providers. Each color band indicates the relative proportion of traces that chose a given international provider as a way of reaching Syrian Telecom's routers. From this, assuming that our traceroute set is large and well-distributed globally (and it is), you can draw some conclusions about the relative weight Syrian Telecom places on each of its provider routes.
Primary transit here was from Tata, Deutsche Telecom, and Turk Telekom, with tiny numbers of paths through other providers. During the first half of the event, all providers' Syrian services were reduced. Tata's and Turk Telekom's shares dropped by about 2/3, proportional to the total number of outaged routes. Deutsche Telekom's service drops by a larger amount, nearly disappearing in the first 12 hours of the outage, despite still being a large part of the inbound available routing.
Halfway through the event, however, more traces start to succeed through Deutsche Telekom, as some routes are partially restored. By 5th June, the trace proportions are basically restored to their relative blend from before the event. It's hard to read, but it suggests to me that this blackout was actively managed by Syrian Telecom, with different groups of networks and different routes coming and going in blocks through the course of the day on Friday, not a simple lights-out event.
You can see that Friday June 10th was nothing like Friday June 3rd. Whatever happened there on the 10th, it did not affect Syria Telecom's provider blend or overall reachability in visible ways. On the 3rd, the government's goal may have been to disrupt online organization of coordinated protests — there aren't that many other explanations for a one-day blackout (videos that couldn't go out to YouTube on Friday, simply went out on Saturday).
And so it goes. The Syrian Internet is, by and large, back online, and has stayed up for a week. This map shows its basic structure and international transit, midway in complexity between Egypt (much larger) and Libya (much smaller). Just a few autonomous systems controlling the transit, just a few dozen networks in all, and yet, it's critical infrastructure for the Syrian people.
Did a day without Internet dampen the protests, or amplify the fury in the streets? One senses that a line has been crossed, and whatever the outcome, the status quo ante solution is now off the table. We can only wait for information to trickle out of Syria, and try to infer the course of events from the available data.
- syria internet blackout
TrackBack URL: www.renesys.com/mt-cgi-bin/mt-tb.cgi/165
About the Renesys Blog
About this Entry
This page contains a single entry by James Cowie published on June 10, 2011 6:00 PM.
World IPv6 Day was the previous entry in this blog.
Libyan Internet Instability is the next entry in this blog.
Find recent content on the main index or look in the archives to find all content.
- The Battle for Tripoli's Internet
- Libyan Internet Instability
- Tracing the Syrian Blackout
- World IPv6 Day
- Syrian Internet Shutdown
- Qwavvis: The Battle for Second
- Level Crossing
- Japan Quake
- What Libya Learned from Egypt
- Libyan Disconnect
- Business (30)
- Economics (12)
- Engineering (65)
- Governance (5)
- Internet (38)
- Meta (2)
- Politics (22)
- Security (15)
- Society (13)
- Alin Popescu (1)
- Bob Fletcher (3)
- Clint Hepner (1)
- James Cowie (26)
- Doug Madory (1)
- Earl Zmijewski (44)
- Martin A. Brown (4)
- Todd Underwood (46)
- Subscribe ATOM Feed
- Subscribe RSS Feed