03.06.24

Threat Actor Exploits Critical TeamCity Vulnerability to Deploy Malware

By Deepwatch Adversary Tactics and Intelligence

Estimated Reading Time: 12 minutes

Key Insights & Determinations

  • Deepwatch has confirmed a threat actor compromised an organization in the Finance and Insurance sector, exploiting the TeamCity vulnerability CVE-2024-27198 to gain initial access and laterally move to other systems. 
  • After gaining initial access, the threat actor created an admin account, implanted a PowerShell script, and deployed an unidentified malware strain. However, no further activity was observed after the malware was implanted. 
  • Rapid7 has discovered and published technical details of two (2) vulnerabilities (CVE-2024-27198 and CVE-2024-27199) impacting JetBrans TeamCity server versions 2023.11.3 and earlier.
  • Within hours of Rapid7’s publication, Proof-of-Concept exploits targeting CVE-2024-27198 were published to GitHub.
  • A threat actor exploiting CVE-2024-27198, the most severe of the two, would allow complete control of the vulnerable TeamCity server, including allowing remote code execution. 
  • Exploitation of CVE-2024-27199 allows an unauthenticated threat actor to modify a limited number of system settings on the server and disclose a limited amount of sensitive information from the server.
  • JetBrains has released a fixed version (2023.11.4). For organizations that can not update at this time, they have also released security patch plugins that fixes these vulnerabilities.

Overview

This week Rapid7 discovered two vulnerabilities, CVE-2024-27198 and CVE-2024-27199, impacting all JetBrains TeamCity server versions 2023.11.3 and earlier. CVE-2024-27198 is an authentication bypass vulnerability in the web component of TeamCity and has a CVSS base score of 9.8, signifying a critical vulnerability. CVE-2024-27199 is also an authentication bypass vulnerability in the web component of TeamCity, with a CVSS base score of 7.3, signifying a high severity vulnerability. On March 4, JetBrains released an advisory and a fixed version of TeamCity. 

During Deepwatch’s investigation and response to these vulnerabilities, we identified that a threat actor compromised an organization in the Finance and Insurance sector by exploiting the TeamCity vulnerability CVE-2024-27198 to gain initial access and laterally move to other systems. After gaining initial access, the threat actor created an admin account, implanted a PowerShell script, and deployed an unidentified malware strain. However, no further activity was observed after the malware was implanted. 

JetBrains TeamCity is a software development lifecycle (SDLC) management platform that helps automate processes for building, testing, and deploying software. It is used by thousands of organizations, such as Citibank, HP, and Ubisoft, to streamline their software development workflows. TeamCity allows developers to manage and monitor their projects, builds, and deployments in a centralized and efficient manner. It provides features such as continuous integration, continuous delivery, and version control integration, making it a valuable tool for software development teams.

Within hours of Rapid7’s publication, Proof-of-Concept (PoC) exploit codes targeting CVE-2024-27198 have been published to GitHub (PoC 1, PoC 2, and PoC 3). One of the PoCs, if successful, adds a new admin user. The other two PoCs, if successful, perform Remote Code Execution (RCE) and include functionality to add an admin user, one of them can also identify potentially vulnerable TeamCity servers.

Analyst Note: At this time, we have not tested the publicly available exploits and cannot confirm if they work as intended.

Exploitation of CVE-2024-27198, the most severe of the two, allows a remote unauthenticated threat actor to take complete control of a vulnerable TeamCity server, including performing remote code execution. Exploitation of CVE-2024-27199 allows an unauthenticated threat actor to modify a limited number of system settings on the server and disclose a limited amount of sensitive information from the server.

According to this Shodan query, there are over 16,000 servers exposed. However, we can not confirm how many of these servers are vulnerable to CVE-2024-27198 and CVE-2024-27199.

Active Exploitation Analysis

The threat actor exploited CVE-2024-27198 in an organization in the Finance and Insurance sector, creating the account “TeamCityService” on a system exposed to the internet. The threat actor then pivoted to a software build environment (BE 1) via functionality within TeamCity. 

After gaining a foothold, the threat actor then implanted at least one malicious executable on BE 1, SysUpdate.exe in the C:\ directory via winpty-agent.exe. However, we could not retrieve this executable. After implanting the executable, the threat actor created a scheduled task to run the executable daily at 10:00 AM on BE 1. Then, the actor created the admin user, “Administralor,” on BE 1 and added the account to 24 groups. After the account creation, the actor dropped the PowerShell script web.ps1, but no evidence of its execution was observed. 

At this point, the threat actor pivoted to another software build environment (BE 2) and implanted the malicious executable SysUpdate.exe. Shortly after implanting the executable, no further malicious activity was observed from the “Administralor” account.

Analyst Note: The implanted SysUpdate.exe was un-signed and its hash was not known to common OSINT sandboxes. Unfortunately, we were unable to retrieve the file from the affected server.

The threat actor also dumped the Local Security Authority (LSA) secrets, and modified the password of the local administrator account multiple times. LSA secrets contain various passwords, such as the user’s text passwords, service account passwords, Internet Explorer passwords, RAS connection passwords, SQL passwords, SYSTEM account passwords, and the private user data like EFS encryption keys.

Vulnerability Analysis

TeamCity servers vulnerable to CVE-2024-27198 expose a web server over HTTP port 8111 by default, but can be configured to run over HTTPS. A threat actor can craft a URL that avoids all authentication checks, allowing endpoints that are intended to be authenticated to be accessed directly by an unauthenticated threat actor. 

A threat actor can exploit this authentication bypass vulnerability in several ways to take control of a vulnerable TeamCity server, and by association, all projects, builds, agents and artifacts associated with the server.

For example, an unauthenticated threat actor can create a new admin user with an actor-controlled password, by targeting the /app/rest/users REST API endpoint. Alternatively, an unauthenticated threat actor can generate a new admin access token. If a threat actor creates a new administrator user account, or generates an administrator access token, they will have full control over the targeted TeamCity server.

The second authentication bypass vulnerability, CVE-2024-27199, allows for a limited number of authenticated endpoints to be reached without authentication. An unauthenticated threat actor can leverage this vulnerability to modify a limited number of system settings on the server and disclose a limited amount of sensitive information from the server.

Rapid7 discovered that by leveraging these endpoints, a threat actor can traverse to an alternative endpoint, without authentication checks being enforced. In Rapid7’s testing, they were able to successfully reach a limited number of JSP pages which leaked information, and several servlet endpoints that both leaked information and allowed for modification of system settings. 

One interesting endpoint a threat actor could target is the /app/https/settings/uploadCertificate endpoint. Reaching this endpoint would allow the threat actor to upload a new HTTPS certificate, as well as change the port number the HTTPS service listens on. 

Reaching the /app/https/settings/uploadCertificate endpoint would allow the threat actor to perform a denial of service against the TeamCity server by either changing the HTTPS port number to a value not expected by clients, or by uploading a certificate that will fail client side validation. Alternatively, a threat actor with a suitable position on the network may be able to perform either eavesdropping or a man-in-the-middle attack on client connections, if the certificate the threat actor uploads (and has a private key for) will be trusted by the clients.

Risk and Impact Assessment

The vulnerabilities CVE-2024-27198 and CVE-2024-27199 present a considerable risk to organizations utilizing JetBrains TeamCity servers. The risk is notably high due to the critical nature of the vulnerabilities – administrative control and system settings modification – which can be exploited without authentication. The publication of Proof-of-Concept (PoC) exploit codes targeting CVE-2024-27198 shortly after the disclosure of Rapid7’s technical publication increases the likelihood of exploitation, as they lower the barrier for threat actors to launch attacks. Given TeamCity’s widespread use for continuous integration and delivery, the vulnerabilities could potentially affect numerous organizations across various sectors.

The impact of an exploitation of these vulnerabilities can be severe for an organization. If threat actors gain administrative control over a TeamCity server, they could manipulate build processes, steal source code, insert malicious code, or disrupt operations, leading to significant operational, financial, and reputational damage. The potential for unauthorized access and control over sensitive data and critical infrastructure elevates the risk of further malicious activities, including data breaches and further lateral movement within the network. Financial implications may include direct losses due to operational downtime, incident response costs, and potential regulatory fines for data breaches. The reputational impact could lead to diminished trust among customers and partners, potentially affecting future business. Given the strategic role of TeamCity servers in software development environments, the cascading effects of such an impact could be widespread, affecting not just the targeted organization but also its customers and partners.

Outlook

The public availability of the exploit codes for CVE-2024-27198 signify a potential uptick in exploitation attempts against JetBrains TeamCity servers. These exploit codes reduce the complexity of launching an attack, making it feasible for a broader range of threat actors, not just the highly skilled ones. Given the critical nature of the vulnerabilities and the swift appearance of PoC codes, we can anticipate an increase in targeted attempts to exploit CVE-2024-27198 in the short to medium term.

As threat actors begin to leverage this vulnerability, we may see an evolution in the threat landscape, with increased incidents of unauthorized access, data breaches, and potentially disruptive attacks on organizations relying on TeamCity servers. The high-profile nature of these vulnerabilities might also encourage other threat actors to discover and exploit similar vulnerabilities in other CI/CD tools, broadening the scope of the threat landscape.

Overall, the outlook for the threat landscape shaped by these events suggests an increase in targeted cyber-attacks exploiting these and potentially other similar vulnerabilities. Organizations should remain vigilant, monitor developments closely, and be prepared to respond swiftly to emerging threats in this evolving landscape. Strengthening security postures and staying informed about the latest threat intelligence will be crucial in mitigating the risks posed by these vulnerabilities.

Actions and Recommendations

JetBrains has released TeamCity version 2023.11.4 which remediates CVE-2024-27198 and CVE-2024-27199. For more details on how to upgrade, please read the JetBrains release blog. 

If you are unable to update your server to version 2023.11.4, JetBrains has also released a security patch plugin so organizations can still patch their vulnerable servers. Organizations can download the appropriate security patch plugin using one of the links below. However, it will only patch the vulnerabilities CVE-2024-27198 and CVE-2024-27199. For information on installing the plugin, see the TeamCity plugin installation instructions.

Security patch plugin: 

We recommend that all customers retrospectively hunt for malicious activity, which may indicate compromise, using the Be On the Lookout (BOLO) guidance provided below:

  • Inbound web requests to impacted appliances following the following pattern: *?jsp=/app/rest/*;*.jsp*
    • Ex: /pwned?jsp=/app/rest/users;.jsp (especially POST requests)
  • Anomalous addition of administrator users on impacted systems
  • Entries in the “teamcity-javaLogging” log file matching the following syntax
    • 26-Feb-2024 07:11:12.794 WARNING [http-nio-8111-exec-1] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI http://192.168.86[.]68:8111/app/rest/users/id:1/tokens/2vrflIqo;.jsp?jsp=/app/rest/users/id%3a1/tokens/2vrflIqo%3b.jsp, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
  • Entries in teamcity-server.log or teamcity-activities.log similar to below indicating possible plugin installation and deletion shortly after (arbitrary code execution, see article for additional artifact locations)
    • [2024-02-26 07:11:13,304] INFO – s.buildServer.ACTIVITIES.AUDIT – plugin_uploaded: Plugin “WYyVNA6r” was updated by “user with id=1” with comment “Plugin was uploaded to C:\ProgramData\JetBrains\TeamCity\plugins\WYyVNA6r.zip”
    • [2024-02-26 07:11:24,506] INFO – s.buildServer.ACTIVITIES.AUDIT – plugin_disable: Plugin “WYyVNA6r” was disabled by “user with id=1”
    • [2024-02-26 07:11:25,683] INFO – s.buildServer.ACTIVITIES.AUDIT – plugin_deleted: Plugin “WYyVNA6r” was deleted by “user with id=1” with comment “Plugin was deleted from C:\ProgramData\JetBrains\TeamCity\plugins\WYyVNA6r.zip”
  • Anomalous usage of the openssl command on impacted appliances
  • Inbound web requests with path segments of “..”, indicative of path traversal, primarily in relation to URI’s within the paths:
    • */res/../*
    • */update/../*
    • */.well-known/acme-challenge/../*
  • Anomalous child processes of java.exe on impacted appliances
    • cmd.exe /c “whoami”
    • cmd.exe /c net group “domain admins” /domain
    • “C:\Windows\system32\net.exe” user administrator Somep@ssword /add
    • systeminfo
    • net user
    • Nltest domain_trusts
    • Net group “domain admins” /domain
    • Net share
    • Route print
    • Winpty-agent.exe
  • Rare EXE files being created/run from the drive root
    • C:\SysUpdate.exe

Observables

The following are computed and atomic indicators that were captured from our internal investigation of active exploitation. The IP addresses were used by the threat actor during authentication events and were observed in the teamcity-auth.log. The file hashes are from the two files the threat actor implanted on compromised endpoints.

Based on Rapid7’s analysis of CVE-2024-27198, potential exploitation attempts to create an access token may leave an artifact in the teamcity-javaLogging log file similar to the following:

26-Feb-2024 07:11:12.794 WARNING [http-nio-8111-exec-1] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI http://192.168.86.68:8111/app/rest/users/id:1/tokens/2vrflIqo;.jsp?jsp=/app/rest/users/id%3a1/tokens/2vrflIqo%3b.jsp, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.

In Rapid7’s above example, the REST API was accessed to create a new administrator access token. In doing so, this log file may contain an entry detailing the URL as processed after the call to ModelAndView.setViewName.

In this example log entry, the URL contains the string ;.jsp as well as a query parameter jsp= which, according to Rapid7, is indicative of an exploitation event. However, the threat actor can include arbitrary characters before the .jsp part, e.g. ;XXX.jsp, and there may be other query parameters present, and in any order.

If the threat actor deleted the access token, the log files teamcity-server.log and the teamcity-activities.log may contain an artifact similar to the one below indicating this. In this example, the artifact shows that the token (“2vrflIqo”) being deleted and should correspond to the token name that was created, which is shown in the warning message in the teamcity-javaLogging log file.

[2024-02-26 07:11:25,702] INFO - s.buildServer.ACTIVITIES.AUDIT - delete_token_for_user: Deleted token "2vrflIqo" for user "user with id=1" by "user with id=1"

If a threat actor exploited CVE-2024-27198 to upload a malicious plugin to achieve arbitrary code execution, the log files teamcity-server.log and the teamcity-activities.log may contain the following artifacts, which, according to Rapid7, indicate a plugin was uploaded, disabled, and subsequently deleted in quick succession, and authenticated with the same user account as that of the initial access token creation.

[2024-02-26 07:11:13,304] INFO - s.buildServer.ACTIVITIES.AUDIT - plugin_uploaded: Plugin "WYyVNA6r" was updated by "user with id=1" with comment "Plugin was uploaded to C:\ProgramData\JetBrains\TeamCity\plugins\WYyVNA6r.zip"

[2024-02-26 07:11:24,506] INFO - s.buildServer.ACTIVITIES.AUDIT - plugin_disable: Plugin "WYyVNA6r" was disabled by "user with id=1"

[2024-02-26 07:11:25,683] INFO - s.buildServer.ACTIVITIES.AUDIT - plugin_deleted: Plugin "WYyVNA6r" was deleted by "user with id=1" with comment "Plugin was deleted from C:\ProgramData\JetBrains\TeamCity\plugins\WYyVNA6r.zip"

Additionally, Rapid7 states that the malicious plugin may have artifacts left in the TeamCity Catalina folder, where the plugin name will form part of the folder name. For example:

Windows: C:\TeamCity\work\Catalina\localhost\ROOT\TC_147512_WYyVNA6r\

Linux: /opt/TeamCity/work/Catalina/localhost/ROOT/TC_147512_WYyVNA6r/

In this example, the plugin is named WYyVNA6r and the 147512 designates the TeamCity server build number.

The webapps plugin folder may also contain a similar artifact. For example: 

Windows: C:\TeamCity\webapps\ROOT\plugins\WYyVNA6r\ 

Linux: /opt/TeamCity/webapps/ROOT/plugins/WYyVNA6r/

The TeamCity data directory may also contain similar artifacts. For example: 

Windows: C:\ProgramData\JetBrains\TeamCity\system\caches\plugins.unpacked\WYyVNA6r\

Linux: /home/teamcity/.BuildServer/system/caches/plugins.unpacked/WYyVNA6r/

Before the threat actor can delete a plugin it must be disabled first, which will leave a permanent artifact in the disabled-plugins.xml configuration file. For example:

Windows: C:\ProgramData\JetBrains\TeamCity\config\disabled-plugins.xml 

Furthermore, the TeamCity administration console has an Audit page that displays activity that has occurred on the server. The deletion of an access token, and the uploading and deletion of a plugin will be captured in the audit log. This audit log is also stored in the internal database data file buildserver.data. 

Finally, to identify unexpected user accounts that may have been created, Rapid7 recommends inspecting the TeamCity administration console’s Audit page for newly created accounts. Both the log files teamcity-server.log and the teamcity-activities.log may contain artifacts indicating a new user account has been created, examples shown below. However, the information logged is insufficient to determine if the created user account is malicious or benign.

[2024-02-26 07:45:06,962] INFO - tbrains.buildServer.ACTIVITIES - New user created: user with id=23

[2024-02-26 07:45:06,962] INFO - s.buildServer.ACTIVITIES.AUDIT - user_create: User "user with id=23" was created by "user with id=23"

Source Material:

↑

Share

LinkedIn Twitter YouTube

Subscribe to the Deepwatch Insights Blog