There were ample signs something big was brewing as all US agencies were issuing warning for Labor Day Weekend. They were too slow to provide useful information, but they were right. On August 25th Atlassian disclosed a vulnerability in Confluence (CVE-2021-26084) and if your organization was running an affected version of Confluence there is no point in assuming you were compromised, just know that you were. Here is what our honey pot revealed.
CVE-2021-26084
Before explaining why you are compromised, and before explaining why it is too late, let me clarify the CVE itself which is materially incorrect. The CVE explicitly states that two conditions had to be true: you must have been running the affected version of Confluence AND “‘Allow people to sign up to create their account’ is enabled.”.
In affected versions of Confluence Server and Data Center, an OGNL injection vulnerability exists that would allow an authenticated user, and in some instances an unauthenticated user, to execute arbitrary code on a Confluence Server or Data Center instance. The vulnerable endpoints can be accessed by a non-administrator user or unauthenticated user if ‘Allow people to sign up to create their account’ is enabled. To check whether this is enabled go to COG > User Management > User Signup Options. The affected versions are before version 6.13.23, from version 6.14.0 before 7.4.11, from version 7.5.0 before 7.11.6, and from version 7.12.0 before 7.12.5.
The second part is wrong. Every instance of Atlassian Confluence was susceptible to a breach. On August 26th Eric Lam, Senior Support Engineer – Confluence and Crowd Server, began clarifying in the tracker that the toggle for the setting was a temporary mitigation measure. Rest assured it did not work and the disclosure was inaccurate – every unpatched instance was vulnerable.
The temporary workaround script must be run even if Confluence Administration > User management > User Signup Options > Allow people to sign up to create their account is not enabled.
There are additional end points identified that still expose Confluence to the CVE-2021-26084 Confluence Server Webwork OGNL injection and applying the workaround script will assist in temporarily mitigating against all known vulnerable end points.
Our Securities Team have since reviewed the wording on the Confluence Security Advisory CVE-2021-26084 – OGNL injection – 2021-08-25 and have removed this ambiguity.
The second key piece here is timing. This did not happen on August 25th. This was tracked by Confluence as of 27/Jul/2021 5:13 AM. While it was found as part of a bug-bounty, the reality here is that we all make assumptions on volume and velocity, and here the CVE marked the terminus for applying patches and not the starting point.
You are compromised.
This vulnerability is low-hanging fruit for a few basic reasons. First, there are 60,000+ Confluence installations in the wild which are not difficult to patch, but they are not easy either. Each upgrade often involves getting a new package, installing it, carrying settings over and restarting. About an hour’s down-time. Occasionally settings don’t port over and you have to restore original config and port individual values into a new config file that has had material changes. Meaning, no one in their right mind just updates Confluence. Even if you read the CVE on August 25th, your organization probably scheduled the patch for some future date — especially since the initial CVE said you had to have “‘Allow people to sign up to create their account” to be at risk. In other words, opportunity was abundant for any malicious organization to look for these instances.
And they sure did – almost immediately. In case you did not know, all that anyone had to do was to scan a series of IPs for Atlassian Confluence responses and attempt to connect. You know what does that really quickly and really effectively? Botnets. Within a couple of days Confluence instances were being detected and compromised by a Monero Mining Botnet and becoming nodes under fully control of malicious actors.
What should you do?
Our guidance on Thursday was “Turn off Confluence and start your incident procedures”, you may try that, but that as two days ago. That’s the least you should do, however now the risks are greater. There was time to transfer the database. There was time to transfer the attachments. There was time to transfer all those files you saved on that server’s hard drive that may have had config info, passwords, or DropBox links that you use for ease.. you know..convenience. There was time for someone to parse through chunks of that unencrypted text data and search for keywords like “password”, “key”, “secret”, and find the nuggets of information that your Wiki may have had. Get third-party help.
How can we be sure?
If you are asking that you’re not paying attention… your Confluence instance is compromised. If it’s still up, look for outbound connections and you may see a sub-process spun off of the Confluence process running a file called “./main” and it will have an outbound connection that is extremely easy to confirm as malicious. If it’s there, don’t be surprised if your Anti-Virus missed it — VirusTotal said “No Security vendors flagged this file as malicious”. Here is what that might look like:
srwxrwxrwx 1 root root 0 Aug 28 12:14 CF_IPC drwxr-xr-x 2 root root 4096 Jun 5 10:27 hsperfdata_root -rwxrwxrwx 1 504 504 14643200 Sep 2 05:54 kinsing -rwxrwxrwx 1 504 504 26800 Sep 2 05:54 libsystem.so -rw------- 1 504 504 0 Sep 2 05:54 linux.lock -rwxrwxrwx 1 504 504 5562368 Sep 2 08:28 main -rw-r----- 1 504 504 5562368 Sep 2 08:16 main.1 -r-xr-xr-x 1 root root 1160 Aug 28 12:14 nj_runInTerminal.sh -rwxrwxrwx 1 504 504 949066 Sep 2 10:23 udp -rw-r----- 1 504 504 949066 Sep 2 10:10 udp.1
Realize that there are two material realities here so far: a payload that is extremely malicious is not visible by anti-malware (we tested with Malwarebytes by the way along with VirusTotal) and a visibly problematic outbound connection. If you have architected an environment properly you may be tempted to breathe easier as you are monitoring CPU usage and Network Outbound and those have not shown spikes. Don’t. The process is throttled to single-digit CPU utilization and 2 Mbps/ outbound. Furthermore, it creates no unusual connections to Confluence to cause spikes in log traffic. In other words CPU, traffic, and log all stay within one standard deviation of norm.
What if you find something suspicious? Well, you have to dig deeper. Did you remember hidden items? Now is the time to do that. When you do a few things should jump out and after a few moments of spelunking you see that “.javae” has a “config.xml” file — that should be suspicious.
drwxr-x--- 2 504 504 4096 Sep 2 05:57 .ICEd-unix drwxrwxrwt 2 root root 4096 Jan 17 2021 .ICE-unix drwxr-x--- 2 504 504 4096 Sep 4 13:08 .javae drwxr-xr-x 2 root root 4096 Sep 2 11:34 .webmin
Here is what that looks like:
drwxr-x--- 2 504 504 4096 Sep 4 13:08 . drwxrwxrwt 7 root root 4096 Sep 2 11:37 .. -rw-r----- 1 504 504 5664 Sep 2 02:43 config.json -rwxr-x--- 1 504 504 2413588 Sep 2 02:39 javae
That leaves one quick look at the config file and sure enough Monero Botnet is confirmed – that’s what “pool.supportxmr.com” is.
"pools": [
{
"algo": null,
"coin": null,
"url": "pool.supportxmr.com:3333",
"user": "HIDDEN",
"pass": "HIDDEN",
"rig-id": null,
"nicehash": false,
"keepalive": false,
"enabled": true,
"tls": false,
"tls-fingerprint": null,
"daemon": false,
"socks5": null,
"self-select": null,
"submit-to-origin": false
}
],
Realize the risks.
The software deployed to your came in on a massive vulnerability that was detected, patched, announced, amended, and slowly trickled out. In that time span the malicious botnets were gobbling it up and using it to their advantage. This is ACT 1 of a multi-act malicious campaign that will play out over the next few days (maybe even this weekend) like this:
- Act 1: take over vulnerable systems and make some money doing Monero mining. We are here today.
- Act 2: spread. Move laterally. Each move exponentially increases mining profits. Some are here today.
- Act 3: ransom. Take data and lock up networks.
- Act 4: more ransom. Did you forget this was a Wiki? How many orgs store sensitive information in this kind of application? Many. There is information there about your org, your customer orgs, your employees, your habits. How about customer passwords? How about all the bits of data you said “we should not store this here” but never thought to act.
Restore Critical Functionality
You may need to get Atlassian Confluence backup. Do it by launching a completely new server and copying your app and your data. Hopefully you designed your system well enough not to have the database there too, and then install Confluence 7.13.0. No maintenance? Buy it because you will need it to activate the license if you were not entitled to the upgrade. Restore your system and then ask yourself: did we detect this fast enough? If you relied on news or notices then you were too slow, and you need to rethink your cybersecurity approach as a whole.
