Ntirety - Newark, DE - Network Event – Incident details

Ntirety has finalized our root cause analysis and will be sending to all customers via ticket and email within the hour. If you do not receive the document please feel free to contact support via phone (866.918.4678), chat or self-service ticket and a technician will provide to you the RCA.

We determined that a Layer 2 broadcast storm within the storage area network caused the service degradation event starting yesterday at approximately 3:00 pm ET. Network switches experienced repeated failures during the incident due to excessive packets per second.  Once the affected network segment was isolated, we restored stability to the customer environments. We are conducting a root cause analysis and will provide our formalized review by the end of the business day on February 3rd.

The platform has shown stability as of 18:30 ET, we continue to monitor the platform for any signs of instability and are working with customers individually for any known issues with their virtual machines. We are still under a root cause review and will share the RCA with all customers when available. Once the root cause is identified we will move forward with a maintenance to bring back redundancy.

All operational teams continue to restore virtual machines that lost connectivity to their guest OS. In parallel, we continue to investigate the main root cause of the switch issue identified, we will not move forward with another emergency change until this is identified connectivity will be single homed.

We have initiated a swap on the identified switch member, and currently monitoring the environment for issues and any further stability concerns. There is a known subset of virtual machines which need manual intervention to regain access to their OS Disk.

After moving forward with a reboot of a host and in addition a member switch, we correlated to the incident. The reboot allowed us to correlate the issue to a member switch, we are currently replacing the switch and will provide an update when replaced and failed over.

We have isolated an issue to a specific host in a unresponsive state. We are initiating an emergency change to address the host and will update the status

We continue to investigate the issue and isolating potential causes, the next update will occur at 15:40 ET.