Following a server level compromise of the Joomla! Extensions Directory (JED), we would like to provide our community a postmortem summary of the events leading to this issue, the response from the Joomla project team members, and a plan of action moving forward to prevent a similar type of issue in the future.
In summary, this was a preventable compromise, and after analysis, we have no reason to believe that any user data has been accessed improperly.
Issue Notification
- At approximately 09:30 UTC on 15 May 2019, a security researcher notified the Joomla Security Strike Team (JSST) that they had discovered an internal Jenkins CI server used by the JED to deploy updates to their live and staging websites and were able to exploit CVE-2018-1000861 on the server, providing a screenshot of a sensitive file as proof of the exploit.
- Upon notification, JSST members worked with JED team members to bring the affected Jenkins system offline and conduct an analysis of whether this server had been compromised in other ways.
Systems Audit
- While investigating the Jenkins server compromise, it was found that a crypto-miner had been installed and was running on the server. A crypto-miner is a software script used to create digital currencies via abuse of server resources (CPU and memory).
- As part of the installed software, a script was found to have been added to the server’s crontab that would attempt to connect to other servers in the local network and install the same miner.
- Since the Jenkins server was used to deploy site updates, the script was able to access the production JED server and install itself there.
- Once it had been discovered on the JED server, steps were taken immediately to bring all services on the affected servers offline and access was restricted to privileged individuals in order to conduct a full root-cause analysis and to begin executing a recovery plan.
- In parallel, the other servers hosting the joomla.org architecture were audited to ensure they had not been compromised as well, and it was determined that only the JED’s servers were affected.
- An analysis was performed on the production JED server to determine the scope of the compromise, including when the server was presumed to be breached and what resources may have been accessed.
- The analysis concluded that the crypto-miner had been installed on the evening of 11 May 2019 and that there was no evidence of improper data access (including access to uploaded extension packages sent to the JED Checker and the site’s database).
- With the analysis concluded, the compromised server was decommissioned with a replacement server activated and a file backup from 10 May 2019 and database from 15 May 2019 restored to the new server.
- The restoration process was completed on 16 May 2019 with the JED team taking action to re-apply pertinent user actions performed between the backup date and the time the JED was discovered to be compromised.
Plan of Action
As a result of the server compromise, several steps are being taken to ensure the security of our servers and our user’s data.
- First, the compromised Jenkins server is scheduled to be permanently decommissioned with the JED migrating to one of the other CI servers used by Joomla in order to eliminate a redundant resource.
- Second, all administrative access (server level passwords and SSH keys) are being reset.
- Third, out of an abundance of caution, all remember me tokens will be invalidated, and all registered users will be required to reset their passwords.
- Lastly, we will be reviewing our internal workflows and procedures and improving our policies and the security features made available to our users across all joomla.org subdomains (such as enabling two-factor authentication on all sites).
Questions and Answers
Q: What was the cause of the compromise?
A: A Jenkins server used to deploy updates to the JED’s production and staging websites, had not been updated to apply a security patch, resulting in the Jenkins server and the JED production server being compromised.
Q: What was the objective of the compromise?
A: According to the analysis, the crypto-miner was installed on the evening of 11 May 2019 and ran until it was detected on 15 May 2019. The crypto-miner abused server resources (CPU and memory) to mine digital currency.
Q: Why wasn’t the security patch applied?
A: The team responsible for keeping the system up to date was working under the assumption that all services running on those two machines, including the Jenkins server, were managed and updated by the hosting company and therefore were not aware that any actions needed to be taken. The Jenkins system, however, was out of scope for the maintenance by the hosting company, leaving it unpatched.
Q: What data has been breached?
A: At this time, we have no evidence to support having had any data breached on the JED’s server.
The exploit payload used in the attack did not have any methods for arbitrary command execution, data exfiltration or spawning a backdoor and therefore simply lacked the code to access the user-related data stored in the systems. Additionally, the amount of data involved in the JED operation has a considerable volume, and we have not detected any actions beyond the regular operation and the mining activity. However, in the unlikely event it is found there was data breached, potentially accessed resources include:
- Uploaded extension packages for JED listings from the local filesystem (used to scan extension code with the JED Checker tool)
- JED user data from the database
- User account data (name, email address, bcrypt hashed password)
- JED support tickets
- JED listing data (extension data, reviews)
Again, at this time we have no evidence to support a data breach, but if this conclusion changes further updates will be provided.
Q: What do I need to do?
A: If you have a user account on the JED website, you will be required to reset your password on the next login. Since user accounts are not shared across joomla.org subdomains, this requirement only affects your JED login. We are also suggesting you change your password on any other websites where you used the same username and password combination, including other joomla.org subdomains. Passwords within Joomla are stored with a bcrypt hash and are not easily guessed or cracked.
Q. Are extension downloads safe?
A: Extension downloads are not served from the JED servers and are therefore not compromised in any way by this breach. All extension downloads and related purchase and/or subscription transactions take place on external services chosen by their developers.
Q. Is my Joomla installation safe?
A: This breach was found in third-party software used by the JED servers, not within Joomla itself.
Q. Has any data been lost due to restoring an old backup?
A: Whilst we have taken steps to ensure that differences in the data have been resolved, the most recent changes (made between approximately 11am UTC and 1pm UTC) were not present in the database backup that was restored. Therefore, we advise that all users login to JED to review any changes that they had made between 10 May and 15 May are still present. This includes submitting new extensions, updating existing extensions, extension reviews, and support tickets.