Last Friday, after Crowdstrike released a bad sensor configuration update that caused widespread crashes of Windows systems. The most visible effects of these crashes appear to have been mitigated. I am sure many IT workers had to spend the weekend remediating the issue.
It is still early regarding the incident response part, but I would like to summarize some of the important facts we know and some lessons learned.
You are likely infected if the CrowdStrike sensor system retrieved updates between 0409 and 0527 UTC on Friday, July 19th. CrowdStrike allows users to configure a sensor update policy, which will delay the update of the sensor software. But the corrupt file was a configuration (“signature”) update, not an update of the sensor itself. Configuration updates are always applied as soon as they are released. Customers do not have an option to delay these updates. Systems crashed because a kernel driver provided by CrowdStrike crashed as it read the malformed configuration file.
Since news of the incident broke, CrowdStrike has been updating and expanding its guidance. Your first stop should be Crowdstrikes “Remediation and Guidance Hub“. It will link to all the resources CrowdStrike has to offer. Yesterday, CrowdStrike announced that they will soon offer a new, accelerated technique for recovery. As I write this, the new technique has not been published. CrowdStrike did provide a new dashboard to affected users to track systems affected by the update.
Microsoft developed a USB solution to simplify the process. To apply the update, systems must be booted from the USB key. However, Bitlocker-encrypted hosts may require a recovery key.
Bitlocker is the major hurdle to a speedy recovery for many affected organizations. Ben Watsons posted on LinkedIn that his organization came up with a way to use a barcode scanner to simplify entering the recovery keys. I do not believe that the related code to create the barcodes is public.
It should be noted that there are some reports of scammers taking advantage of the incident. I reported on Friday about some phishing attempts and domains registered to take advantage of the incident. So far, we have not received a sample of a phishing e-mail, just reports that they had been seen. These phishing and malware emails may affect organizations not directly affected by the CrowdStrike problem. The extensive news coverage, often called a “Windows Problem”, may prompt users into installing malicious files.
If you are affected: Only use tools provided by trustworthy sources. Refer to CrowdStrike’s advice for guidance, and be careful with advice from others (including me 🙂 ). Do not make far-reaching infrastructure changes before the incident is completely understood, and plan any changes carefully. This isn’t the time to “rip out” CrowdStrike without first carefully evaluating alternatives. It may take a few weeks for CrowdStrike to completely understand what happened. Resiliency isn’t just about avoiding outages. A big part is how to deal with outages that may happen. If you are not in the midst of recovering from CrowdStrike, Think about how you would deal with all your Windows Server (or Workstations) going down. How would you continue operations? Do you know where your Bitlocker recovery keys are?
If you are interested in recent domains registered to take advantage of the incident: Try our API. For example:
https://isc.sans.edu/api/recentdomains/today/crowdstrike?json
Instead of “crowdstrike,” you may use other keywords or replace ‘today’ with a date in YYYY-MM-DD format. A suspect domain registered today: crowdstrike-fix.zip.
—
Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu
Twitter|
(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.