Although fiber cuts happen daily all over the world, last weeks cuts in the Bay Area suddenly became interesting news. Why? Not because of the massive outages (seriously, there have been incidents with underseas cables that have left entire countries offline for days or weeks. The reason this became newsworthy was the fact that these were deliberate cuts:

As you may have heard on the news, some time during the early morning hours today an AT&T employee accessed a manhole between Redwood City and San Carlos CA. and cut all fiber links. This has affected Telekenex and all major carriers (AT&T, Verizon MCI/Sprint, Level 3, Abovenet Communications and others). Services throughout the Bay Area have been affected, as well as links that were serviced from this location (Seattle being one of them)

This sparked a short discussion on the nanog mailing list about protecting physical infrastructure. It doesn’t really matter how well you protect your data centers and offices, if the actual fiber running between them is out in the open without any protection whatsoever. The thread starts here, and is well worth reading.

As noted in other places, the fiber paths are well-known, and sometimes even public information. Although there have been attempts to hide this from the public eye (anyone remember Sean Gorman?), it’s difficult to keep this a total secret. There are simply too many people involved in fiber rollouts, and especially in rural areas most people living in the area will know where the fiber runs are. And of course there’s always the call-before-you-dig option!

So given that hiding the location of major communication lines is hard to do, some people are pushing for extra protection of the actual access to it. In this particular case, the cables were accessed through a manhole; there are companies that provide “safe” manhole covers, such as MBSS, so it might be possible to prevent this kind of event.

manhole_security

But as many people have noted, these protections won’t add a lot of extra security. Quoting from the aforementioned thread on the nanog list:

Matthew Moyle-Croft has this observation about locks on manholes:
Some people do lock their vaults/pits/manholes.  But, to be honest, I’m not sure it helps a lot.  How many passersby would stop someone appearing to be in a phone company/telco high-vis vest using bolt cutters – telling them the lock had seized?

JC Dill adds that there is no financial incentive to make additional security worthwile:
My guess is that it is probably less expensive in the long run to leave them unprotected and just fix the problems when they occur than to try to “secure” the vaults and deal with the costs and extended outage delays when access it “secured” and it takes longer to get into a vault to fix things.


Finally Valdis Kletnieks chimes in with a very accurate observation:
The alarm that goes off saying the lid got opened is only 2 minutes before the big red alarm that says you just lost 5 OC-768s.  So the link is *still* going to drop even as you’re on the 911 call to try to explain to them where your manhole is, the cops *still* won’t catch anybody (the perps may be gone before you hang up on the 911 call), and you’re taking 2 minutes off a 10-hour outage.

And, even if all manholes are secured, there is still the issue of rural areas; anyone with a backhoe can severely damage vital infrastructure. This will even work in most cities; it is not unusual to open up streets for “maintenance”.

backhoe
So in this case the obvious conclusion is that physical security for large fiber paths is virtually impossible; the best way to protect against these events is making sure you use multiple, diverse paths so you can route around outages.