Internet2 Monitoring Hurricane Harvey
August 31, 2017 Update
We learned this morning that the Liberty, TX amp site is currently inundated with water and running on generator. This is shown on the map below as LIBE.
The specific text from the Level(3) portal states: “Field Services were able to obtain access to the failure site in Liberty, TX using a boat. Flood waters have breached the site, and identified approximately six inches of water is inside the site, impacting all equipment on the lower shelves. Field Services reports that commercial power to the site is currently down, but a backup generator is functioning with adequate fuel remaining, and battery backup should the generator fail. Field Services personnel report that once the waters subside, a fuel truck will be dispatched to refuel the generator and maintain power until commercial power can be restored to the site.”
At this point, the largest operational risk remains the loss of amp sites on the two southern paths. Analysis of our current traffic patterns shows that we have sufficient capacity for reroute conditions should these paths become inoperable. We are continuing to monitor the situation closely.
August 30, 2017 Update
At this point, the flood risk seems to be reduced for our facilities. No further information has come from Level(3) or our partners on the ground. Floodwaters in the vicinity of the Hardy St PoP continue to recede.
In nearby webcams, you can actually start to discern ground features which were previously submerged.
August 28, 2017 Update
Internet2 expresses our deep condolences to the people impacted by the devastation caused by Hurricane Harvey. In particular, Houston, Texas is continuing to see record rainfalls with 24 inches of rain falling since Hurricane Harvey made landfall. Many of the areas highways and bayous are severely flooded. The National Weather Service released a recent report that an additional 26 inches may fall through Friday of this week.
Internet2 and the Internet2 NOC continue to actively monitor the situation in Houston and in the surrounding areas very closely. Internet2 has two operations sites in the Houston area located near I-10 and I-69; one located near the Beltway and I-45. These sites maintain a combination of optical transport and routing/switching equipment. Additionally, several amp sites (ILAs) are within the footprint of the storm. These sites provide important services to our member community as well as backup paths for nearly all our east-west community traffic. As of this morning, Monday, August 28, all PoPs and ILAs remain in service on utility power. Internet2 remains committed to work with our partners to restore any impacted services or potential outages if they occur.
As the weather can be unpredictable, we will continue to monitor the situation closely to determine what additional impacts may arise to our campus members and regional network partners in the area. We will continue to provide updates later this week if necessary.
We hope all people in the area affected by the storm, especially our colleagues and their families, remain safe, and will keep them in our thoughts.
August 25, 2017 Update
Internet2 has been monitoring Hurricane Harvey and issued an initial note to the community yesterday evening (see below). The National Hurricane Center (NHC) forecast indicates that potential flooding is expected across the middle and upper Texas coast with heavy rainfall of 15 to 25 inches - with isolated amounts as high as 35 inches - from Friday, August 25th through next Wednesday, August 30th. Although forecast tracks beyond three days can change dramatically, recent NHC models show the hurricane shifting course after landfall and moving along the gulf coast towards Louisiana; that would potentially bring the level of rain up through Houston and potentially into Louisiana. Regardless of whether this "meandering" storm model comes to pass, Harvey is nonetheless expected to be a major hurricane at landfall, bringing life-threatening storm surge, rainfall, and wind hazards.
In addition to the risk to human life and critical utilities, extraordinary rain of this nature often impacts the telecommunications infrastructure in these affected areas, causing disruption to national, regional and campus networks in a broad area. In past hurricanes, Internet2 has experienced damage ranging from no impact to more severe multi-day outages when flooding precluded delivery of fuel to Level(3) sites or flooded operational sites themselves. Additionally, Members have also previously experienced severe issues in low lying areas of campuses and regional network PoPs, particularly in coastal areas. While both the Internet2 network and our regional partners work to deliver highly resilient services, events like this can have unexpected impacts that may require attention.
The Internet2 NOC is actively monitoring/managing network changes in anticipation of potential outages and reduced resiliency along the gulf coast. The map below demonstrates where peak rainfall potentially overlays with Internet2 and it's members infrastructure in the landfall zone. Considering the strength of this storm and the amount of projected rainfall, it is possible we will see service impacts along our southern paths which may result in substantial traffic transiting through Chicago and other northern cities. Internet2 will work diligently with our partners to restore any impacted services as soon as it is safe to do so. We also will make best effort to offer mutual aid to campus and regional networks as well as national network partners to help them with any impact on their networks. Safety and well-being will always remain the most important factors in these situations.
It has been an extraordinary show from nature this week, first with the eclipse and now with this dangerous hurricane. For those in its path, be safe.