¿How to Recover Your Windows Operating System from the CrowdStrike Failure?

Introduction

A faulty update to CrowdStrike’s Falcon sensor, released on July 18, 2024, has caused widespread Blue Screen of Death (BSOD) errors on Windows systems globally. This issue has significantly impacted various organizations, including airlines, banks, and media outlets, severely disrupting their operations. In this article, we detail the impact of the failure and provide specific recommendations for recovering affected systems and restoring services.

Details of the failure

On July 18, 2024, a faulty update to CrowdStrike's Falcon sensor triggered a surge in Windows services, leading to a global failure that requires specific actions to resolve. Symptoms of the issue include BSOD errors related to the Falcon sensor.

Additional details

• Windows hosts that were not affected do not require any action, as the problematic channel file has been reverted.

• Hosts connecting after 0527 UTC are not affected.

• Windows 7/2008 R2 systems and Mac or Linux-based systems are not impacted.

• The problematic channel file is "C-00000291*.sys" with a timestamp of 0409 UTC, while the reverted version has a timestamp of 0527 UTC or later.

Global impact

1. Operational disruptions: Various sectors reported significant disruptions. Airlines experienced delays and cancellations, banks faced issues with their transactional systems, and media outlets encountered technical difficulties affecting content broadcasting.

2. Data loss: BSOD errors resulted in the loss of unsaved data, impacting both personal and corporate information. Data recovery and system restoration have become complex and costly processes.

3. Reputation damage: Trust in CrowdStrike’s security solutions has been compromised, prompting organizations to reassess their cybersecurity strategies and consider additional measures to prevent future incidents.
   

Recovery recommendations

1. Patch update

CrowdStrike has released a corrected version of the patch. Organizations should ensure all systems are updated with this version to prevent further BSOD errors.

2. Manual removal of the faulty driver

For affected systems, it is recommended to:

• Boot Windows in Safe Mode or Windows Recovery Environment (by pressing F8 during startup).

• Use a wired network connection and Safe Mode with Networking to aid in the fix.

• Navigate to %WINDIR%\System32\drivers\CrowdStrike.

• Locate and delete the file "C-00000291*.sys" with the 0409 UTC timestamp.

• Restart the host normally. For Bitlocker-encrypted hosts, a recovery key will be required.

3. System restoration

If manual removal is not feasible, use system restore points to revert the operating system to a state before the faulty patch was installed.

4. Alternative solutions for cloud and Virtual Environments

Option 1:

• Disconnect the operating system volume from the affected virtual server.

• Create a new recovery state or backup of the volume.

• Attach the volume to a new virtual server.

• Navigate to %WINDIR%\System32\drivers\CrowdStrike and delete the problematic file.

• Reconnect the volume to the affected virtual server.

Option 2:

Revert to a state prior to 0409 UTC.

5. Alternative solution in Azure

• Log in to the Azure console and select the affected VM.

• Access the Serial Console and execute commands to configure Safe Mode.

• Use the bcdedit command to set up Safe Mode boot and revert changes after the fix.

6. Automated solution via Group Policy

A GPO can be configured to run a script in Safe Mode:

Creation of the PowerShell Script:

# This script deletes the problematic CrowdStrike driver file causing BSODs and reverts Safe Mode

$filePath = "C:\Windows\System32\drivers\C-
00000291*.sys"
$files = Get-ChildItem -Path $filePath -ErrorAction
SilentlyContinue

foreach ($file in $files) {
	try {
		Remove-Item -Path $file.FullName -Force
		Write-Output "Deleted: $($file.FullName)"
	} catch {
	Write-Output "Failed to delete: $($file.FullName)"
	}
}

# Revert Safe Mode Boot after Fix
bcdedit /deletevalue {current} safeboot

Creation of the GPO:

• Create a GPO to apply the deletion script and configure Safe Mode boot.

Conclusion

The incident involving the faulty CrowdStrike update highlights the need for a proactive and resilient security posture. Organizations must take immediate action to mitigate the impact and strengthen their defenses against future threats.

At DeltaWits, we are committed to providing the necessary support to overcome such challenges and help organizations maintain operational continuity and confidence in their security systems.