Handling compromised Google Cloud credentials

Google Cloud credentials control access to your resources hosted on Google Cloud. To help keep your data secure and protected from attackers, you must handle your credentials with utmost care.

We recommend that you protect all of your Google Cloud credentials from unintended access. These credentials include but are not limited to the following:

Google Cloud CLI credentials are stored in the user's home directory. You can list them in Google Cloud CLI using the gcloud auth list command. Application Default Credentials are stored on the developer's workstation. Browser cookes are browser-specific, but are typically stored on the developer's workstation.

If you suspect that any of your credentials have been compromised, you must take immediate action to limit the impact of the compromise on your Google Cloud account.

Monitor for credential compromise

To monitor for potential compromise, consider the following:

  • Monitor for suspicious account activity such as privilege escalation and multiple account creations. Monitor for these activities using Cloud Audit Logs and Event Threat Detection. Configure alerts based on administrator activity in Compute Engine audit logs and Google Kubernetes Engine (GKE) audit logs. Use Event Threat Detection to identify threats that are based on administrator activities, Groups changes, and Identity and Access Management (IAM) permission changes.

  • Monitor user logins in Google Workspace and Cloud Identity. To better track issues, consider exporting the logs to Cloud Logging.

  • Monitor for secrets in your code repositories, using tools such as secret scanning.

  • Monitor for anomalies in service account key usage using Cloud Monitoring.

Ensure that your security operations center (SOC) is notified promptly. You can integrate Security Command Center with your SIEM, export logs from Cloud Logging to your SIEM, or import logs into Chronicle for further analysis.

Ensure that your SOC has the playbooks, tools, and access that are required to respond quickly to suspected credential compromise.

Protect your Google Cloud resources from a compromised credential

Complete the steps in the following sections as soon as you can to help protect your resources if you suspect a credential is compromised.

Revoke and reissue credentials

If you suspect a credential is compromised, revoke and re-issue it. Proceed carefully to ensure you do not suffer a service outage as a result of revoking credentials.

In general, to reissue credentials, you generate a new credential, push it to all services and users that need it, and then revoke the old credential.

The following sections provide specific instructions for each type of credential.

Replace a service account key

  1. In the Google Cloud console, go to the Service accounts page.

    Go to Service accounts

  2. Locate the affected service account.

  3. Create a new key for the service account.

  4. Push the new key to all the locations in which the old key was in use.

  5. Delete the old key.

For more information, see Creating and managing service accounts.

Regenerate API keys

  1. In the Google Cloud console, go to the Credentials page.

    Go to Credentials

  2. Create a new API key using the Create credentials button. Configure the new key the same as the compromised API key. The restrictions on the API key must match, otherwise you might suffer an outage.

  3. Push the API key to all locations in which the old key was in use.

  4. Delete the old key.

For more information, see Using API keys.

Reset an OAuth2 client ID secret

Changing a client ID secret will cause a temporary outage while the secret is rotated.

  1. In the Google Cloud console, go to the Credentials page.

    Go to Credentials

  2. Select the compromised OAuth2 client ID and edit it.

  3. Click Reset Secret.

  4. Push the new secret to your application.

For more information, see Setting up OAuth 2.0 and Using OAuth 2.0 to access Google APIs.

Remove Google Cloud CLI credentials as an administrator

As a Google Workspace administrator, remove access to Google Cloud CLI from the user's list of connected apps. For more information, see View and remove access to third-party applications.

When the user accesses Google Cloud CLI again, it will automatically ask them to re-authorize the application.

Remove Google Cloud CLI credentials as a user

  1. Open the list of apps with access to your Google Account.

  2. Remove Google Cloud CLI from the list of connected apps.

When you access Google Cloud CLI again, it will automatically ask you to re-authorize the application.

Revoke Application Default Credentials as an administrator

If you suspect that an Application Default Credential is compromised, you can revoke it. This procedure can cause a temporary outage until the credentials file is recreated.

As a Google Workspace administrator, remove access to the Google Auth Library from the user's list of connected apps. For more information, see View and remove access to third-party applications.

Revoke Application Default Credentials as a user

If you suspect that an Application Default Credential that you created is compromised, you can revoke it. This procedure can cause a temporary outage until the credentials file is recreated. This procedure can only be completed by the owner of the compromised credential.

  1. Install and initialize the Google Cloud CLI, if you haven't already.

  2. Authorize gcloud CLI with your user identity, not with a service account:

     gcloud auth login
    

    For more information, [Authorizing the gcloud CLI], see (/sdk/docs/authorizing).

  3. Revoke the credentials:

      gcloud auth application-default revoke
    
  4. Optionally, delete the application_default_credentials.json file. The location depends on your operating system:

    • Linux, macOS: $HOME/.config/gcloud/
    • Windows: %APPDATA%\gcloud\
  5. Recreate the credentials file:

     gcloud auth application-default login
    

Invalidate browser cookies as an administrator

If you suspect browser cookies are compromised, Google Workspace administrators can sign a user out of their account.

In addition, immediately force a password change.

These actions invalidate all existing cookies, and the user is asked to log in again.

Invalidate browser cookies as a user

If you suspect browser cookies are compromised, sign out of your Google Account and change your password immediately.

These actions invalidate all your existing cookies. The next time you access Google Cloud, you must log in again.

Look for unauthorized access and resources

After you revoke compromised credentials and restored your service, review all access to your Google Cloud resources.

  1. Examine your audit logs in the Google Cloud console.

    Go to Logs Explorer

  2. Search all potentially affected resources, and make sure that all account activity (especially related to the compromised credentials) are as expected.

Delete all unauthorized resources

Make sure that there are no unexpected resources, such as VMs, App Engine apps, service accounts, Cloud Storage buckets, and so forth, that the compromised credential could access.

After you are satisfied that you have identified all unauthorized resources, you can choose to delete these resources immediately. This is especially important for Compute Engine resources, because attackers can use compromised accounts to exfiltrate data or otherwise compromise your production systems.

Alternatively, you can try to isolate unauthorized resources to allow your own forensics teams to perform additional analysis.

Contact Cloud Customer Care

For help with finding the Google Cloud logs and tools that you require for your investigation and mitigation steps, contact Customer Care and open a support case.

Best practices to avoid compromised credentials

This section describes best practices that you can implement to help you avoid compromised credentials.

Separate credentials from code

Manage and store your credentials separately from your source code. It is extremely common to accidentally push both credentials and source code to a source management site like GitHub, which makes your credentials vulnerable to attack.

If you are using GitHub or other public repository, you can implement tools such as secret scanning, which warns you about exposed secrets in your GitHub repositories. To stop keys from being committed to your GitHub repositories, consider using tools such as git-secrets.

Use secret management solutions such as Secret Manager and Hashicorp Vault to store your secrets, rotate them regularly, and apply least privilege.

Implement service account best practices

To help protect service accounts, review the best practices for working with service accounts.

Limit session lengths

To force periodic re-authentication, limit the time that sessions remain active for Google and Google Cloud accounts. For more information, see the following:

Use VPC Service Controls to limit access

To limit the impact of compromised credentials, create service perimeters using VPC Service Controls. When you configure VPC Service Controls, resources inside the perimeter can only communicate with other resources inside the perimeter.