Following a server level compromise of the Joomla! Extensions Directory (JED), we would like to provide our community a postmortem summary of the events leading to this issue, the response from the Joomla project team members, and a plan of action moving forward to prevent a similar type of issue in the future.
In summary, this was a preventable compromise, and after analysis, we have no reason to believe that any user data has been accessed improperly.
Issue Notification
- At approximately 09:30 UTC on 15 May 2019, a security researcher notified the Joomla Security Strike Team (JSST) that they had discovered an internal Jenkins CI server used by the JED to deploy updates to their live and staging websites and were able to exploit CVE-2018-1000861 on the server, providing a screenshot of a sensitive file as proof of the exploit.
- Upon notification, JSST members worked with JED team members to bring the affected Jenkins system offline and conduct an analysis of whether this server had been compromised in other ways.
Systems Audit
- While investigating the Jenkins server compromise, it was found that a crypto-miner had been installed and was running on the server. A crypto-miner is a software script used to create digital currencies via abuse of server resources (CPU and memory).
- As part of the installed software, a script was found to have been added to the server’s crontab that would attempt to connect to other servers in the local network and install the same miner.
- Since the Jenkins server was used to deploy site updates, the script was able to access the production JED server and install itself there.
- Once it had been discovered on the JED server, steps were taken immediately to bring all services on the affected servers offline and access was restricted to privileged individuals in order to conduct a full root-cause analysis and to begin executing a recovery plan.
- In parallel, the other servers hosting the joomla.org architecture were audited to ensure they had not been compromised as well, and it was determined that only the JED’s servers were affected.
- An analysis was performed on the production JED server to determine the scope of the compromise, including when the server was presumed to be breached and what resources may have been accessed.
- The analysis concluded that the crypto-miner had been installed on the evening of 11 May 2019 and that there was no evidence of improper data access (including access to uploaded extension packages sent to the JED Checker and the site’s database).
- With the analysis concluded, the compromised server was decommissioned with a replacement server activated and a file backup from 10 May 2019 and database from 15 May 2019 restored to the new server.
- The restoration process was completed on 16 May 2019 with the JED team taking action to re-apply pertinent user actions performed between the backup date and the time the JED was discovered to be compromised.
Plan of Action
As a result of the server compromise, several steps are being taken to ensure the security of our servers and our user’s data.
- First, the compromised Jenkins server is scheduled to be permanently decommissioned with the JED migrating to one of the other CI servers used by Joomla in order to eliminate a redundant resource.
- Second, all administrative access (server level passwords and SSH keys) are being reset.
- Third, out of an abundance of caution, all remember me tokens will be invalidated, and all registered users will be required to reset their passwords.
- Lastly, we will be reviewing our internal workflows and procedures and improving our policies and the security features made available to our users across all joomla.org subdomains (such as enabling two-factor authentication on all sites).
Questions and Answers
Q: What was the cause of the compromise?
A: A Jenkins server used to deploy updates to the JED’s production and staging websites, had not been updated to apply a security patch, resulting in the Jenkins server and the JED production server being compromised.
Q: What was the objective of the compromise?
A: According to the analysis, the crypto-miner was installed on the evening of 11 May 2019 and ran until it was detected on 15 May 2019. The crypto-miner abused server resources (CPU and memory) to mine digital currency.
