July 21, 2024 Advisory: CrowdStrike Falcon Sensor Issues

With recovery activity in full swing across the globe and many reaching out for quick fixes and easy workarounds, the hacking community have pounced on the opportunity to Masquerade as Genuine Help and carry out their malicious agenda on unsuspecting users seeking help.

Wipro SOC Tiger Team along with the CRS Threat Intel Team has been monitoring this situation closely and have created a security Threat Intelligence Advisory with IOCs to help prevent such hackers taking advantage of this situation.

• Threat Intel Advisory to proactively prevent Cyber Attacks on unsuspecting users seeking help for the BSOD issue caused by CrowdStrike’s faulty update on Windows machines – SOC Tiger Team and CRS Threat Intel Team
• Threat Intel Advisory 2.0 to proactively prevent Cyber Attacks on unsuspecting users seeking help for the BSOD issue caused by CrowdStrike’s faulty update on Windows machines – SOC Tiger Team and CRS Threat Intel Team

While this is not a cyberattack, be vigilant of Phishing attempts pretending to be Crowdstrike tech support or other platform tech support services. Let’s not feed into the hackers who will be looking to maximize this disruption to their advantage.

A recent update to the CrowdStrike Falcon Sensor has caused significant disruptions, leading to Windows machines crashing and displaying the Blue Screen of Death (BSOD). This issue has affected numerous organizations globally, impacting critical services and operations.

Details

• Root Cause:
  • CrowdStrike Falcon System Driver Update: The issue stemmed from an update to CrowdStrike Falcon Sensor software’s driver, which impacted Windows machines and caused BSOD. However, it’s important to note that this was not a cyberattack.
• Channel Driver File Details:
  • The reverted (good) version: Channel file “C-00000291*.sys” with a timestamp of 0527 UTC or later.
  • The problematic version: Channel file “C-00000291*.sys” with a timestamp of 0409 UTC.
• Impact: Affected systems are unable to boot normally, leading to widespread operational disruptions. Customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. Various versions of the Falcon Sensor, including (but not limited to) version 6.58, have been reported to cause issues.

Workaround Steps

For Individual Hosts:

1. Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again:
2. Boot Windows into Safe Mode or the Windows Recovery Environment.
   • Navigate to C:\Windows\System32\drivers\CrowdStrike directory.
3. Locate the file matching “C-00000291*.sys” and delete it.
4. Boot the host normally.

Note: BitLocker-encrypted hosts may require a recovery key.

For Public Cloud or Similar Environments (Including Virtual):

1. Detach the operating system disk volume from the impacted virtual server.
2. Create a snapshot or backup of the disk volume as a precaution.
3. Attach/mount the volume to a new virtual server.
   • Navigate to C:\Windows\System32\drivers\CrowdStrike directory.
4. Locate the file matching “C-00000291*.sys” and delete it.
5. Detach the volume from the new virtual server.
6. Reattach the fixed volume to the impacted virtual server.

Option 2: Roll back to a snapshot before 0409 UTC.

For Azure via Serial Console:

1. Log in to the Azure console.
2. Go to Virtual Machines and select the VM.
3. Click “Connect” > “More ways to Connect” > “Serial Console.”
4. Once SAC has loaded, type in ‘cmd’ and press enter.
5. Type the following commands:
   • ch -si 1
   • bcdedit /set {current} safeboot minimal
   • bcdedit /set {current} safeboot network
6. Restart the VM.

Optional: To confirm the boot state, run the command: wmic COMPUTERSYSTEM GET BootupState.

Reference Link: https://azure.status.microsoft/en-gb/status/

For AWS EC2 Instances:

1. Reboot the instance (may allow for the Falcon agent to be updated).
2. If not successful, follow these steps:
   • Create a snapshot of the EBS root volume.
   • Create a new EBS Volume from the snapshot in the same availability zone.
   • Launch a new Windows instance using a different version of Windows.
   • Attach the EBS volume from step (2) to the new instance as a data volume.
   • Navigate to \windows\system32\drivers\CrowdStrike\ and delete “C00000291*.sys.”
   • Detach the EBS volume from the new instance.
   • Create a snapshot of the detached EBS volume.
   • Replace the root volume of the original instance with the new snapshot.
   • Start the original instance.
3. EC2 Instance Recovery: Customers can safely relaunch their EC2 instances from a snapshot or image taken before 9:30 PM PDT. The problematic update that caused the CrowdStrike Falcon agent issue is no longer being automatically applied, ensuring that the relaunched instance won’t be affected by the issue.
4. For Amazon Workspace:
   • Reboot Affected Workspaces: Consider rebooting the impacted Amazon Workspaces. While this step may recover the instance for some users, it does not guarantee success universally.
   • Restore from Backup: Alternatively, restoring the affected Workspaces to a recent backup is recommended. This approach ensures that you revert to a known good state.

Reference Link : https://health.aws.amazon.com/health/status

For Google:

https://status.cloud.google.com/incidents/DK3LfKowzJPpZq4Q9YqP

Learnings from Other Accounts

Observations from other accounts dealing with remote workers include:

Deskside Engineer Assistance: Some organizations have found success by having deskside engineers connect with end users via Teams calls on mobile devices. During these video calls, the engineer shares a one-time BitLocker recovery key and guides the end user through the recovery process. While this approach is effective, ensure that it aligns with your organization’s security practices.

Patch Information

CrowdStrike has identified the problematic update and rolled it back. The channel file “C-00000291*.sys” with a timestamp of 0527 UTC or later is the reverted (good) version. Conversely, the channel file with an earlier timestamp will be the problematic one. Engineering teams are actively working on a permanent fix. Users are advised to monitor official CrowdStrike communication channels for further updates and detailed patch information.