This blog provides the latest update on the CrowdStrike-Microsoft outage, which has caused a global IT disruption affecting thousands of companies worldwide. The fallout has been extensive, impacting everything from airports and banks to some government entities.
Microsoft has acknowledged the issue and posted updates on their Azure status page. They’ve received reports of successful recovery from customers who attempted multiple virtual machine (VM) restarts. This is promising news for server administrators dealing with non-loading apps and services likely tied to downed VMs.
In the Azure Portal, attempting to restart affected VMs can be done either directly or via command line/Azure Shell. Reports indicate that several reboots—sometimes up to 15—may be required. While it seems odd that simple reboots might resolve the issue, feedback suggests it’s an effective troubleshooting step at this stage.
Additional Recovery Steps:
Backup Restoration:
Customers are advised to restore from a backup made before 1900 UTC on July 18th, which is when the problematic patch was deployed. If you have such a backup, restoring it could be a straightforward solution.
OS Disk Repair:
For those unable to restore from backup, Microsoft provides instructions for repairing the OS disk offline. This process involves deleting a specific file in the CrowdStrike directory:
Dealing with Encrypted Drives:
If your drives are encrypted, additional steps to unlock the disks for offline repair are necessary. This typically involves using BitLocker or similar encryption tools. After addressing the encryption, follow the same process of detaching and reattaching the disk to the original VM.
This outage has significantly impacted VMs globally within the Azure environment, particularly those running Windows and CrowdStrike together.
Here are the primary steps to consider for recovery:
- Reboot affected VMs multiple times.
- Delete the specified file in the CrowdStrike folder if reboots don’t work.
- Restore from a backup made before the critical patch date.
- Perform offline OS disk repairs, especially if using encrypted drives.
We will continue to monitor the situation and provide updates as more information becomes available. The global IT landscape relies heavily on these systems, and it’s crucial to get everything back up and running swiftly.
