VMware ESXi is an enterprise-class, type-1 hypervisor that allows organizations to deploy and host several of their virtualized systems running multiple operating systems on a single physical server. If ransomware successfully infected a Windows computer, a single machine is affected but if it infects a VMware ESXi, it hits a whole group of servers at once rendering all the virtual machines (VMs) unusable. Thus, from an attacker’s perspective, infesting this hypervisor is like “hitting n birds with one stone”. It can wreak havoc in any enterprise environment.
On February 3, 2023, the CERT-FR and the web hosting provider OVHcloud issued warnings regarding a global ransomware campaign targeting VMware ESXi servers. This ransomware is named “ESXiArgs”. It's been confirmed that the ransomware attack is not exploiting any zero days but rather the “two-year-old” CVE-2021-21974.
OVHcloud has stated that the vulnerability in OpenSLP was exploited as an initial compromise vector but the involvement of CVE-2021-21974 (a heap-overflow vulnerability where a malicious actor residing within the same network segment can trigger this issue in OpenSLP service resulting in remote code execution) has not been confirmed. The involvement of CVE-2022-31696, CVE-2022-31697, CVE-2022-31698, CVE-2022-31699, CVE-2021-21995, CVE-2021-21974, CVE-2020-3992, and CVE-2019-5544 has also been speculated by the media.
In October 2022, Juniper detected a custom python backdoor “
vmtools.py" implanted on a VMware ESXi virtualization server. It was a cross-platform python script that got executed on startup and launched a simple web server accepting password-protected POST requests which could either run arbitrary remote commands with results as a webpage or launch a reverse shell on any port as per the need. There is mention of “
vmtools.py" in the “encrypt.sh” script so this backdoor might also have a certain role in the infection routine.
On the first day of detection
Approximately 120 ESXi servers were encrypted. But, more than 4000 hosts worldwide have been affected as of now. On 8th Feb, Ransomwhere mentioned, “4 total payments have been made to date for a total of $88k.
Instructions were left by the ransomware on how to restore the files which also contains a bitcoin address where the victims should make the payments. Ransomwhere created a github page containing a list of ESXi ransomware payment addresses.
Here is a sample image of encrypted files we obtained from the bleeping computer forum.
Encrypted files on the victim's machine
VMware released a patch in their advisory on 23rd Feb 2021 regarding multiple CVEs including this one. Most organizations should have already patched their ESXi servers but, patching ESXi can be challenging and requires downtime. So, some organizations may not have updated to a fixed version.
As of 8th February, the second ESXiArgs ransomware wave started. It is now encrypting more extensive amounts of data, making it much harder, if not impossible, to recover encrypted VMware ESXi virtual machines. Preliminary reports indicated OpenSLP vulnerability to be the initial compromise vector but a comment that was posted in the forum stated that the SLP service was disabled on their devices and was still breached and encrypted. Also, the ransom note has been changed:
In this new ransom note, the bitcoin address has been removed, and now advises the victims to contact the adversaries on TOX with the provided ID.
ESXi hypervisor is too sensitive to be exposed to the internet and can pose significant security risks. But still “85,546” ESXi servers are exposed to the internet as of 20th Feb 2023.
Here is an overview of the data depicting the overall infections so far as of Feb 20, 2023:
ESXiArgs Ransomware Tracker
ESXi versions 7.x prior to ESXi70U1c-17325551
ESXi versions 6.7.x prior to ESXi670-202102401-SG
ESXi versions 6.5.x prior to ESXi650-202102101-SG
Initial compromise vector: CVE-2021-21974 (not confirmed)
The public key used for encryption along with other necessary artifacts are dropped in /tmp/ directory.
The encryption process targets files with “.vmdk”, “.vmx”, “.vmxf”, “.vmsd”, “.vmsn”, “.vswp”, “.vmss”, “.nvram”, ”*.vmem” extension and leaves the .args extension after encryption.
The malware tries shutting down VMs by killing the VMX process to unlock the files.
The malware creates “argsfile” to store arguments passed to the “encrypt” binary (number of MB to skip, number of MB in encryption block, file size)
No data exfiltration observed so far.
Some cases show partially failed encryptions, allowing the victim to recover data with ease.
BleepingComputer has created a dedicated forum where people are reporting their experiences and receiving help to recover their machines. So, you can also find some relevant information here. Stay updated with this page.
For the first wave, security researcher Enes Sonmez created a recovery guide that allows admins to rebuild their virtual machines and recover their data for free.
Also, CISA has released a recovery script to allow organizations to attempt recovery of virtual machines affected by the ESXiArgs ransomware attacks.
Quick Mitigations for ESXiArgs ransomware
Check your version of ESXi and stay updated with the main VMSA page to apply the latest security patch. Here are the latest available supported releases of vSphere components. You can look at some tips regarding patching here.
Ensure your data is backed up and only necessary services are active. Make sure they are filtered with ACL to only trusted IP addresses.
Keep monitoring your system for abnormal behaviors and keep necessary defenses in place. One of our security researchers Bhabesh Raj had also published a sigma rule for detecting exploitation of VSphere Remote Code Execution vulnerability as described in CVE-2021-21972 so it can also be used.
Disable the OpenSLP service on the server if not used, you can follow this link.
Do not expose your ESXi instances directly to the internet if at all possible, place filters and additional security controls in front of them and review those controls for effectiveness.
Moving on to the technical analysis part, this ransomware drops “encrypt” an encryption program, and “encrypt.sh” the associated shell script and other artifacts in the victim machine. Upon inspecting the shell script we have observed the following.
First, it creates a staging directory and places its artifacts in that directory. It then lists out the virtual machine's configuration files and modifies the path to the virtual disk and swap files appending “1” before the file extensions and force-terminates all running virtual machines.
Then it moves towards encryption. It searches for the file with the specific extensions i.e: “.vmdk”, “.vmx”, “.vmxf”, “.vmsd”, “.vmsn”, “.vswp”, “.vmss”, “.nvram”, and “.vmem” and then calculates their size so that the files smaller than 128MB are encrypted in a single step and those larger than 128MB are encrypted in multiple steps using the binary executable “encrypt” that the ransomware dropped with an argument file “public.pem” in “/tmp/” directory.
In the new variant everything is the same, however only the block for 'size_step=0' in “encrypt.sh” has been removed and size_step is set to 1 so that the encryptor alternates between encrypting and skipping 1 MB of data.
After the encryption process, “index.html” and “message of the day(motd)” are replaced by the ransom note, and all the files with “.log” extension are removed.
Then, It lists and checks the installed versions of VMware. If the number of builds with version “7.x” not equal to 0, it removes the last and the first 8 lines of this “/var/spool/cron/crontabs/root” and sets its modification time same as "/usr/lib/vmware/busybox/bin/busybox" to remove its tracks. However, if it is equal to 0, it removes only the first line of the “/sbin/hostd-probe” .
In the final step, It modifies and removes multiple files and their artifacts to evade detection. Among those files ”/store/packages/vmtools.py,” is the python backdoor, mentioned in this blog earlier.
After the clean-up operations, It makes sure the changes in the configuration files will be intact and deletes itself. Finally, it starts the SSH service, possibly in the event that the threat actor wants to return to the server.
Removing artifacts from the victim machine
Hunting ESXiArg with Logpoint
There has been almost no talks on its detection so far. While detecting is a first step, it is also a somewhat passive approach. Organizations need to adopt a more proactive posture in cyber threat hunting. For which Logpoint comes into play. We have tried to simulate the attack using “VMware ESXi 6.7.0” and used its log files for hunting this ransomware.
Log sources required:
- Web Proxy
- VMware ESXi
The initial access vector seems to be the exploitation of a heap-overflow vulnerability resulting in remote code execution but the specific CVE is not confirmed yet. So, Analysts should be on the lookout for the exploitation of potential attack vectors. The following query can help defenders to identify the exploitation of a VSphere Remote Code Execution vulnerability.
Also, look out for suspicious failed remote login attempts. Account locking is supported for access through SSH and through the vSphere Web Services SDK. By default, a maximum of five failed attempts is allowed before the account is locked. The account is unlocked after 15 minutes by default. However, an attacker might persistently try multiple times to log in, so analysts should monitor this as well.
As previously mentioned, after gaining access to the system, adversaries are seen dropping their artifacts and running malicious scripts.
On Thursday, September 29th, Mandiant published information on malware they discovered in the wild that leverages unsigned VIBs to install backdoors on a compromised ESXi host. Analysts are suggested to look for the command line used to force the installation of the malicious VIB and the corresponding command line used to verify the VIB signature in ESXi.
Responding to ESXiArgs Ransomware using Logpoint SOAR
Logpoint SOAR is by far one of the most useful tools for organizations to investigate and respond to ransomware attacks. With available playbooks, we can respond to the situation and prevent further damage to the system. Among many, the following playbook can be useful in instant response to this ransomware.
This playbook checks if any IP, domain, URL, or host exists in a list of IoCs, blocks them, and adds them to the blocked list preventing damage to the system.
Tips for Ransomware Resilience with VMware Cloud Infrastructure
Always be prepared. Assuming that you will be attacked ensures they're covered in your organization’s disaster recovery & business continuity planning.
Use centralized logging of ESXi environments. Ensure all ESXi host and vCenter Server logs are being forwarded to the organization’s SIEM solution like Logpoint to provide visibility into security events.
Classify your systems according to their business criticality. It will help you assign correct security controls to systems, assess dependencies, and prioritize work during an incident.
Simulate these kinds of incidents every now and then, and make your staff aware of such situations.
Patching the vCenter server might affect the management interface. So, systems that connect to vSphere should be set to automatically reconnect and resume operations when vCenter Server becomes available again.
Manage the infrastructures with a separate dedicated admin account not used for regular tasks so that, if the attackers use stolen credentials, it shows up in the failed login attempt logs.
Avoid using authorization groups because attackers can add themselves to the group after they compromise the system and log in to systems they should not otherwise have access to.
Ensure that only specific authorized devices can access the management interfaces and systems. Also, avoid/restrict causal or malicious access to it and monitor it as well.
Manage vSphere through vCenter Server with Role-Based Access Control (RBAC) permission model and by default disable SSH and enable normal Lockdown mode in ESXi.
Use multifactor authentication and time-based access where possible.
Keep your backup systems separate from corporate identity systems. It should be an island, extremely isolated, managed by few staff, monitored heavily, but also protected against outside influence from centralized monitoring and management systems.
Use immutable backups and/or backup “air gaps” where possible.
Enable the Secureboot feature in ESXi to mitigate the risk of malicious actors persisting on a compromised ESXi host via malicious VIB installation.
You can also follow these guidelines from VMware to harden your system.
Designing Infrastructure to Defeat Ransomware.
Ransomware: Defense in Depth with VMware (white paper).
VMware Cloud DR Ransomware Recovery Guide.
Cloud Infrastructure Security Configuration Guides.
Protecting vSphere From Specialized Malware Guide.
Always keep your systems up to date and apply security patches ASAP. However, if the patch cannot be applied immediately, make sure to apply temporary mitigations.
For the present issue: Disable SLP service, ensure the ESXi hypervisor is not exposed to the public internet if possible, or make sure that it is behind VPN. A recent report suggests the new variant of the Royal ransomware is also targeting ESXi servers so, stay extra cautious. If you have been infected by the first wave, solutions are available which are mentioned above. But for the second wave, there is a minimal chance that you can recover from it.
However, If you follow the hunting and responding portion mentioned above, you can almost prevent the attack or detect it in time. It is important to adopt a multi-layered defense approach for your system. Logpoint’s Converged SIEM is a key solution to monitor suspicious activities in your environment and detect any chained events of an ongoing attack through out-of-the-box alert rules. Further, the SOAR component can be actively leveraged for automated responses and to prevent any extensive damage to the system environment.
Be secure, be safe.