What We Know So Far
Early in the morning on 7/19/24, CrowdStrike applied an update to their platform that has caused widespread Blue Screens of Death (BSODs) on Windows hosts. The update impacted multiple sensor versions. The update has since been reverted; however, machines that had the update applied will need to follow remediation steps performed on them to resolve the crash conditions.
The following tenants are in scope: EU-1, US-1, US-2 and US-GOV-1.
We will send additional updates / learnings as we work with our customers, CrowdStrike, and Deepwatch Operations & Engineering.
Impact to Deepwatch Managed Cyber Services
Deepwatch Operations is not currently experiencing any outage due to this event. Core MDR monitoring is not impacted. Visibility via CrowdStrike sensors are impacted while CrowdStrike mitigation steps are being rolled out in a controlled fashion as we support our customers.
Current CrowdStrike Sensor/Channel File Remediation Steps
Current remediation steps for each host. No current solution to do this without manually (or via remote systems management) going to each host.
- Boot Windows into Safe Mode without Networking or the Windows Recovery Environment
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Locate the file matching “C-00000291*.sys”, and delete it.
- Boot the host normally.
Troubleshooting and Key Learnings So Far
- SafeMode without Networking: You must boot the machine into SafeMode without Networking. As of now, if you try to boot into SafeMode with networking it will cause a boot loop. This is causing some machines to need to be reimaged
- To boot Windows 10 or 11 in Safe Mode, you can try these steps:
- Restart your computer
- At the sign-in screen, hold down the Shift key and select Power > Restart
- After the computer restarts, select Troubleshoot > Advanced options > Startup Settings > Restart
- After the computer restarts again, a list of options should appear
- Select 4 or F4 to start in Safe Mode
- If you need to use the internet, select 5 or F5 for Safe Mode with Networking
- BitLocker. If BitLocker is enabled in the organization. In order to apply the remediation steps, the organization must apply the remediation steps to the BitLocker key server first. Then get the BitLocker key for each host and move on to applying the above steps to each host.
- USB Device Control. If your organization has USB Device Control enabled, you may need to modify the policy in order to assist with the remediation steps.
- Admin Rights: IT teams will need admin rights to login in locally and delete the file needed to recover.
- Domain Controllers: Replication can be broken as a result of this issue.
Active Directory extends the single-master model found in earlier versions of Windows to include multiple roles, and the ability to transfer roles to any DC in the enterprise. Because an Active Directory role isn’t bound to a single DC, it’s referred to as an FSMO role. Typically, an FSMO role ownership is executed only when the domain controller has replicated the naming context (NC) where the ownership is stored since the Directory Service started. Ensure the FSMO roles are assigned to the appropriate Domain Controllers once the patch has been applied. Currently in Windows there are five FSMO roles:- Schema master
- Domain naming master
- RID master
- PDC emulator
- Infrastructure master
Additional Resources
- CrowdStrike Tech Alert:
- AWS Advisory:
- Azure Advisory:
Customer Communications – Update on Lessons Learned and What We Know
Post 8am EST email distribution to customers
Update 2 to CrowdStrike Issue Flash
3:45pm
Reboot the host to give it an opportunity to download the reverted channel file. If the host reboots and is working properly, then no further action is required. If the host crashes again, then attempt manual remediation.
- Boot Windows into Safe Mode or the Windows Recovery Environment
- Note: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.
- Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
- Note: On WinRE/WinPE, navigate to the Windows\System32\drivers\CrowdStrike directory of the OS volume
- Locate the file matching “C-00000291*.sys”, and delete it.
- Boot the host normally.
7/20/2024 12:50 UTC Update 3 Monitoring Dashboards
As of 7/20/2024 01:00 UTC CrowdStrike has released a dashboard that can be used to assist organizations in tracking potentially still impacted systems. The Dashboards can be found at the via the following links;
US1: https://falcon.crowdstrike.com/investigate/search/custom-dashboards
US2: https://falcon.us-2.crowdstrike.com/investigate/search/custom-dashboards
EU1: https://falcon.eu-1.crowdstrike.com/investigate/search/custom-dashboards
GOV: https://falcon.laggar.gcw.crowdstrike.com/investigate/search/custom-dashboards
Once at the dashboard screen search for a dashboard called
Hosts_possibly_impacted_by_windows_crashes
Once in the dashboard set the status dropdown to check and click apply.
The results of status CHECK means CrowdStrike believes these machines to be currently impacted and remediation steps are needed.
The results of status VERIFY mean CrowdStrikes believes these machines could still be impacted by the BSOD and would warrant further investigation to assess if remediation steps are needed on that host.
[NEW] 7/23/2024 17:57 UTC Update 4: CrowdStrike File Quarantine Update
As of 2024-07-23 1757 UTC, Crowdstrike has added the affected channel file to the quarantine file list for all customers.
The channel file responsible for system crashes on Friday, July 19, 2024 beginning at 04:09 UTC was identified and deprecated on operational systems. When deprecation occurs, a new file is deployed, but the old file can remain in the sensor’s directory.
Out of an abundance of caution, and to prevent Windows systems from further disruption, the impacted version of the channel file was added to Falcon’s known-bad list in the CrowdStrike Cloud.
No sensor updates, new channel files, or code was deployed from the CrowdStrike Cloud.
For operational machines, this is a hygiene action. For impacted systems with a strong network connectivity, this action could also result in the automatic recovery of systems in a boot loop.
This was configured in US-1, US-2, and EU on July 23, 2024.
Continue to use the available dashboards to assess impact and remediation efforts.
The logic in the dashboards has been updated to reflect a confidence level of the impacted systems as follows:
- DOWN: a high confidence assessment where remediation is likely to be required
- Endpoint has channel file version of 0 and has not checked-in after impact window.
- Endpoint received channel file during impacted window, but endpoint has NOT checked-in after impact window.
- VERIFY: a low to medium confidence assessment
- Endpoint received channel file during impact window and has checked-in after impact window.
- RECOVERY_LIKELY: a medium confidence assessment
- Endpoint received channel file during impact window and has checked-in after impact window with a total reported uptime of 5-10 hours.
- RECOVERY_VERY_LIKELY: a medium to high confidence assessment
- Endpoint received channel file during impact window and has checked-in after impact window a total reported uptime of 10-20 hours.
- UNKNOWN: there is not enough available data to form an assessment
- Cannot determine endpoint status based on available telemetry.
- OK: a high confidence assessment
- Endpoint running version of Falcon sensor that is not impacted.
- Endpoint has latest channel file and is operational.
- Endpoint was offline and did not receive channel file during impact window.
- Endpoint was online and did not receive channel file during impact window.
Additional details regarding use of the dashboards provided by CrowdStrike is available here:
https://supportportal.crowdstrike.com/s/article/ka16T000001tm1eQAA
↑
Share