The dark cloud around GCP service accounts

Step 2: Narrow down the search space

With so many questions, where do we begin? This is where the science of research becomes more of an art. There are many reasons this might be occurring. How can we effectively narrow it down? Furthermore, how can we know we are correct in what we ultimately find? Let’s first consider the core question we want to answer: why does this service account still have access even though I deleted its service account key? With that in mind, let’s attempt to reason about what services of the over 200 that GCP offers are relevant to the questions we want to answer. We need to determine the relevant components of granting a service account access to a GCP environment.

First we need to understand how a request from a service account is evaluated and ultimately granted or denied. At a high level, it follows these three steps:

1. Authentication – verifying the identity of the requestor
2. Authorization – ensuring the requestor is allowed to perform the action
3. Validation – checking the action against policy/verification of the request

Since the service account still has access, it must be passing all three of these checks. Service accounts and service account keys are part of the IAM service in GCP. This is where I decided to focus my research.

Step 3: Investigate

Now that we have narrowed down the search space to be more manageable, we can begin investigating. There are a number of ways a service account can gain access to a GCP environment: workload identity federation, service account impersonation, attaching a service account, and service account keys, to name a few. In the workshop I was using service account keys, so this is where I focused my initial investigation. I first asked myself: “How does a service account key grant the service account access?” I knew from experience teaching the workshop that you can run the following command to create a service account key file:

Here is what a service account key file looks like:

I then went through and tried to figure out as much as I could about each field to determine what role they played in providing the service account access to the GCP environment. Not all fields were well documented, or documented at all.

When considering the various fields in the file I decided that the private_key field is one of the most interesting. It refers to an x.509 certificate used for verification. I figured this must be part of the authentication portion of the flow. I was also aware of the ability to sign JWT (JSON Web Tokens) using x.509 certificates. I also noticed that when I visited the URL listed in the client_x509_cert_url, it had the private_key_id listed there next to a public x.509 certificate.

Step 4: Test the hypothesis

My first thought was that the creation of the access file was granting me access to the environment. However, this turned out to be false. When you run the gcloud command to create the service account key, you don’t immediately have access to the environment. You can test this by running the command to create the key then trying to run a command as the service account. It will not work. That then led me to look at the next command in the sequence:

This command is what actually configures the gcloud tool to access the GCP environment as the service account. It also needs the access file as input, so it must be using that information to set up access. I now have a bunch of new questions to answer:

1. What does that command do that allows me to access GCP as the service account?
2. How does it generate credentials to access the environment?
3. What information is it using from the access_key.json file to do that?
4. Where is the credential stored on my local system?

Luckily, the gcloud tool is open source… sort of. It’s written in Python, and when you install it you can browse all the Python files to see what they do. After some rummaging through the source code I determined that it was using the info in the access_key.json file I had created to sign a JWT and then use that to get an access token. It then stores this token locally at ~/.config/gcloud/access_tokens.db. This is a sqlite database that stores access tokens used by gcloud. To see if this really was the access token the service account was using I ran the following command to list all of the tokens it had stored:

I then found the entry that had the name of my service account. I ran a simple curl command and sure enough, it worked!

I then verified in GCP’s Logs Explorer that the request was made from the service account. You can see in the authenticationInfo section a reference to the key id that was used to authenticate the service account:

So it seems that the key file is used to sign a JWT that is then exchanged for an access token. That access token is stored in a local database. The access token is all that is really needed to make a request to the GCP environment. So in theory I should just be able to revoke that access token and I am good to go, right? Well, it’s not that simple. Here are some of the things I tried to revoke the token:

1. I looked in the google-auth library to see if there was a way to revoke a token
2. I looked at the token info endpoint documentation
3. I looked at service account key credential documentation
4. I tried deleting the service account key with gcloud iam service-accounts keys delete <key-id>
5. I tried gcloud auth revoke <svc_acct> This just deletes the entry in ~/.config/gcloud/credentials.db and ~/.config/gcloud/access_tokens.db. If you have the raw access token you can still create a new key by saving the access token and running a command like the following:

Finally, I found this in the gcloud documentation:

Turns out, GCP service account tokens are not revocable. This is why deleting the service account key does nothing!

Step 5: Summarize your findings

At this point I was confident that I understood the process used to get an access token for the service account. I also felt confident that there wasn’t a way to revoke a service account’s access token.

Now it’s time to bring these insights from the realm of research to practical application.

The service account key file is used to generate an access token

The service account key file contains all the info needed to get an access token. This means that anyone with this file has the ability to obtain an access token as the service account. This is the main reason why Google discourages the use of service account keys.

The access token is stored locally on the machine

The access token is trivially accessible for those on the machine. This means it could be targeted by adversaries who gain access to a victim’s computer. With a lifetime of one hour, the tokens are short lived. The service account key credentials are also on the machine and can be used to generate new access tokens. They are located at ~/.config/gcloud/credentials.db.

The access token cannot be revoked

This means that if an access token is compromised, there is not a good path to remediation aside from disabling the service account and waiting up to an hour for the token to expire. Disabling a service account will likely cause a serious disruption to business operations as the services that leverage that account will not be able to function. This is amplified if the service account happens to be a default service account.

Next steps

Consider potential adversary tradecraft

Putting this research in the context of what is already known about service accounts raises several security concerns. Certain services in GCP, such as Compute Engine and App Engine, generate default service accounts. Up until May 2024 service accounts in an organization were assigned the primitive editor role by default. This role allows service accounts to create service account keys. Why is this important? Well, this means that deleting the service account key and waiting for the access token to expire is no longer an option. The service account can periodically check if it still has its service account key. If not, it can just create a new one for itself and authenticate with that.

This means the only option to remove an adversary’s access to your account is to disable/delete the service account. Unfortunately, that is usually not a practical solution. Here is what Google has to say about it:

“You can disable or delete this service account from your project, but doing so might cause any applications that depend on the service account’s credentials to fail.”

Take steps to mitigate this behavior

So what can you do to protect your GCP environment? Be sure to narrowly scope the permissions of a service account. It should only have the minimum set of permissions needed to perform its actions within the GCP environment. Also, as Google recommends, don’t use service account keys if at all possible. You can enforce this using constraints at the organization level. Below is Google’s guidance regarding proper use of service accounts:

“Depending on your organization policy configuration, the default service account might automatically be granted the Editor role on your project. We strongly recommend that you disable the automatic role grant by enforcing the iam.automaticIamGrantsForDefaultServiceAccounts organization policy constraint. If you created your organization after May 3, 2024, this constraint is enforced by default.
If you disable the automatic role grant, you must decide which roles to grant to the default service accounts, and then grant these roles yourself.

If the default service account already has the Editor role, we recommend that you replace the Editor role with less permissive roles. To safely modify the service account’s roles, use Policy Simulator to see the impact of the change, and then grant and revoke the appropriate roles.“

Create detection logic for this behavior

It is possible to detect this type of behavior by looking for a service account creating a service account key for itself. This is highly unusual behavior as most of the time a service account key is created by some other granting authority such as a different service account or a user. A search of this behavior across all of our data yielded no instances of this occurring.

Collapse the box below for a sample log of this activity occurring. You can see that the email address in the authenticationInfo section is the same as in the request.name field.