Crowdstrike Outage: How ETS Responded and Recovered

On the night of July 18, 2024, a routine software update from Crowdstrike, the cybersecurity tool used across ASU, led to a significant disruption for Windows computers and servers within the Fulton Schools of Engineering (FSE) and across the broader ASU network. This issue resulted in system crashes and BitLocker Recovery prompts, causing considerable downtime.

Understanding the Incident and the Response

At around 9:47 PM, the ETS Systems Orchestration team detected the issue, identifying the Crowdstrike update as the cause of the widespread system failures. Working closely with ASU’s Enterprise Technology (ET) team, ETS quickly mobilized a coordinated response to contain the problem. The immediate priority was to stop the spread of the faulty update by removing it from affected servers and systems.

Throughout the night and into the following day, individuals worked diligently to restore critical systems. Efforts included both on-campus and remote interventions to ensure that classroom and lab disruptions were minimized. Collaboration with IT teams across individual schools within FSE played a vital role in bringing systems back online swiftly and effectively.

Innovative Solutions and Collaborative Recovery

To accelerate the recovery process, ETS implemented innovative tools, including custom USB boot devices and a Raspberry Pi-based recovery system. The Raspberry Pi devices were configured to automate the process of booting affected systems and applying necessary fixes, reducing recovery times from hours to just minutes per system. By the early morning of July 19, essential services were largely restored, with continued efforts ensuring that all systems were back online as quickly as possible. The success of this recovery was due to the collective efforts of technologists across FSE, and the support of ASU’s broader technology community.

Looking Forward

The Crowdstrike outage underscored the importance of robust communication protocols and effective incident management strategies. As we move forward, ETS is committed to refining these processes and working collaboratively to strengthen our systems against future challenges.

If you experienced issues during this time, or if you’d like to express your thanks, please reach out through ServiceNow. Your feedback is invaluable as we continue to support the FSE community.