Attacker’s Footprints

Stackoverflow recounts an attacker's persistence over weeks of exploration and sequential acquisition of increased privilege. In an interesting twist they could also track the ongoing use of their Q&A resources to understand the accumulation of skills. post

In order to understand how the privilege escalation and subsequent exfiltration of source code occurred, we needed to be able to trace the attacker’s accesses to our sites prior to the culmination of the attack.

Fortunately, we have a database containing a log of all traffic to our publicly accessible properties—this proved invaluable in identifying activity associated with the attacker. Using the account identifier that had been escalated, we were able to use the IP address and other identifying information to correlate traffic to a candidate set of rows.

Coupled with other information from our customer support team and various other sources of log data, we came up with a timeline of events. This is quite detailed but it’s here because we’d like to bring attention to the amount of time the attacker took to understand our infrastructure and gradually escalate their privilege level to the point at which they could exfiltrate our source code.

(The first points of attack reveal intent and maybe checking to see if there are easy targets: aiming for build and source control, "notably SRE room"... and a social engineering attempt to get source code.)

Tuesday April 30th, 2019 The attacker starts probing our infrastructure, in particular parts of our build/source control systems and web servers hosting some of our development environments.

Wednesday May 1st The attacker continues probing our public network and attempts to access employee-only rooms in Stack Exchange Chat—notably our SRE room. They get “access denied.” Additionally a person claiming to be one of our Enterprise customers submits a support request to obtain a copy of source code for auditing purposes. This request is rejected because we don’t give out source code and, additionally, the email cannot be verified as coming from one of our customers. It is flagged for further investigation by our support team.

(the story continues describing an attacker with clear patience and wide probing for vulnerabilities... pivots to customer support attacks, discovers private urls in public github repos, and so on.)

(eventually the attacker finds credentials for Team City and shifts attacks over there. Fun example of strange loop pattern, the attacker searches stack exchange to learn configuration options to continue attacking Team City.)

Wednesday May 8th Attacker immediately logs into TeamCity and continues browsing the administrative sections of the site. They stumble across a diagnostics section that allows browsing of the build server’s file system—this yields a plaintext SSH key used by build agents to obtain source code from GitHub Enterprise. Within minutes, this was used to clone several key repositories (gathered from VCS roots configured in TeamCity).

Saturday May 11th This is the day that the attacker manages to escalate their privileges across the Stack Exchange Network. Once again, the attacker pulls the latest copies of source code and immediately starts to investigate build logs and configurations. (...attacker configures a build in TeamCity that uses a public github gist to inject a privilege escalation. Using the Stack Exchange build system to attack Stack Exchange)

Sunday May 12th Shortly after execution of the SQL, we were notified of the odd activity by the community and our incident response team started investigating.

.

Deploy pipeline attacks are similar problem to Supply Chain Hacks.

The sprawling attack also motivates another reason I want Vault Auth and Better Kubernetes Secrets.