Skip to content

    Incident Response in Google Cloud: Forensic Artifacts

    February 2, 2023

    Key Takeaways

    • Forensic data across Google Cloud can logically be organized into three categories: Identity Management, Google Workspace Apps, and Google Cloud Platform (GCP). Each category can be further broken down into four subcategories: Configurations, Logs, Reports, and Alerts.
    • During triage, prioritize the following evidence sources when performing incident response against Google Workspace:
      • Alert Center alerts > Admin reports > Identity logs > Application logs > Application data
    • During triage, prioritize the following evidence sources when performing incident response against Google Cloud Platform:
      • Alert Center alerts > Identity logs > Security and Platform logs > Service and Resource data

    Introduction

    In our previous blog, we provided a foundational view on identity management in Google Cloud, which helped in understanding that evidence sources may vary depending on whether the cloud environment consists of GCP, Google Workspace, or Google Cloud Identity. We also discussed how these larger components are interrelated and from an incident response perspective, have the capacity to detrimentally affect each other. To build upon this foundation, this article will examine forensic artifacts available and provide recommendations for triage and prioritization.

    Creating a Mental Model: Forensic Artifacts in Google Cloud

    Mental models serve to simplify complex scenarios and create an approach to reasoning through them. Because there are many artifacts spread throughout Google Cloud across a multitude of services and locations, we will categorize everything for ease of tracking. While there are limited or paid-tier services (e.g., Security Investigation Tool in Google Workspace and Security Command Center in GCP) that perform security-related functions useful during incident response, we will focus on the most widely available evidence sources.

    A basic tenant of incident response is to understand in all data sources where potential anomalous or malicious activity can occur. For this reason, Google Cloud data sources have been broken down into three categories: identity management for the overall Google Cloud, applications specific to Workspace, and resources and services within GCP. Each of these categories are broken down further into subcategories:

    • Configuration: service, application, and access settings and configurations.
    • Logs: track administrative actions and access across Google Cloud resources.
    • Reports: statistical information, presented in pre-built graphs and tables.
    • Alerts: provides Google pre-configured and custom alerting.

    Blog2_Figure1Figure 1: Mental model for evidence available in Google Cloud

     

    Google Cloud forensic data can be accessed through the Admin Console or via APIs for Google Cloud identity management and Google Workspace, and through Google Cloud console, Google Cloud CLI (GCloud), or via APIs for GCP.

    We will first begin by recommending evidence prioritization via triage, and subsequently examine the data sources to better understand their role in threat hunting and incident response.

     

    Triage in Google Cloud

    Triage is the process of assigning priority to sources of evidence and affected assets during a security incident based on efficacy and risk. The ability to quickly identify, prioritize, and resolve security events is critical when managing or responding to incidents within a Google Cloud environment.

    In the sections below, we have designated artifact priority based on the likelihood of availability and utility of data captured. The sections provide an example for triage in Workspace and GCP separately, while considering identity-based evidence across both. The steps of the triage may vary based on the incident details, priority, and resources.

     

    Triage in Google Workspace

    figure02_Ver3

    Figure 2: Triage path for incident response in Google Workspace

     

    Triage in Workspace involves its managed identities and their interactions with the productivity applications. With an examination of the Alert Center in the Admin console, we can quickly identify anomalous activity associated with specific identities or applications.
    Once a preliminary review of alerts has been completed, Admin reports can provide a high-level overview of productivity application use and potential abuse. For example, we can identify potential phishing campaigns launched from Workspace via Gmail reports, specifically through analysis of outbound email delivery spikes. The identity and application logs can then be used to pivot from compromised accounts to discover additional events and indicators of compromise (IOCs). Once the review of alerts, reports, and logs has been concluded, a deeper dive into specific application data can begin (e.g., gathering email data) if necessary.

     

    Triage in Google Cloud Platform

    figure03_ver3

    Figure 3: Triage path for incident response in GCP

    Triage in GCP involves the domain’s identities and their interactions with its services and resources. As the Alert Center is available in the Admin Console by default, its alerts can provide easily identifiable IOCs to pivot from. After reviewing alerts, the identity logs (User log, Admin log, SAML log, etc.) will help to pinpoint any abnormal access into the Google Cloud domain. Once the affected identities have been scoped, identifying actions taken within GCP itself is observable in the Security and Platform logs. More specifically, begin by examining the default enabled Admin Activity audit log for GCP resource modifications and the Data Access audit log (if available) for user-driven resource access. Once these initial artifacts have been exhausted and a better grasp of the security incident has been achieved, the investigation can be conducted effectively by targeting precise services or resources. 

    After exploring the triage methodology, we will now cover each described data source to understand what it consists of and its forensic value for both incident response and threat hunting.

    Alerts

    Identity Management & Workspace – Alert Center Alerts

    Alert Center alerts provide pre-configured and custom alerting on potential issues within a Google Cloud domain, either by Google Cloud’s identity management or Workspace Apps. Pre-configured alerts are enabled by default and should be one of the first sources of evidence examined when dealing with a security incident. These alerts capture abnormal user, administrative, and device events. Many security incidents that result in large-scale compromise can often be prevented or mitigated by reviewing security alerts as they occur.

    The Alert Center is located in the Admin Console in “Security tab -> Alert center” and retains data for approximately ten years. Although a curated selection is described below, visit Google documentation for a full list of alerts.

     

    Name

    Description

    General alerts

    Security and privacy issues, government-backed attacks

    User alerts

    Suspicious login, user granted admin privilege, user suspended, password change, suspicious programmatic login

    Administrative alerts 

    Super admin password reset, primary admin changed, SSO profile added, domain data export initiated 

    Gmail alerts

    Employee spoofing, malware detected post-delivery, suspicious messages, user-reported phishing

    Custom alerts

    Manually configured activity, reporting, and DLP alerts

     

    Reports

    Identity Management & Workspace – Admin Reports

    Admin Reports summarize user security-oriented settings, application activity, and billing costs in a statistics form. Admin Reports consist of data from Google Cloud’s identity management and Workspace Apps. The statistics report, which are either presented as graphs or as tables provide easily digestible output that can be utilized as a quick method to identify anomalous activity.

    Admin Reports are found in the Admin Console in “Reporting -> Reports” and are retained for six months. Although there are more reports available, we can focus on the ones that can be more easily utilized for security purposes.

     

    Category 

    Name

    Description

     

    Highlights

    Highlights

    Captures Workspace App use, user status, storage quota, document visibility, and security metrics

     

    Apps Reports

    Accounts

    Captures organizational security-related setting metrics

     

    Aggregate Reports

    Captures metrics across all Google Apps

     

    Drive

    Captures active user, file, and share metrics

     

    Gmail

    Captures email metrics (e.g., emails sent/received)

     

    User Reports

    Apps Usage

    Captures user-based Google Workspace Apps usage metrics (e.g., last Gmail access time through web or legacy protocols, storage quotas, file creation); Captured Workspace Apps depend on Google Workspace license

     

    Security

    Captures security-related setting status per user

     

    Apps Usage

    Captures user-based Google Workspace Apps usage metrics (e.g., last Gmail access time through web or legacy protocols, storage quotas, file creation); Captured Workspace Apps depend on Google Workspace license

     

    Although many of the metrics and trends covered by Admin Reports do not provide practical evidence for the incident response process – Gmail report, user-based Apps Usage report, and cost report can be helpful in identifying irregular application usage. For example, the Gmail app report will track the frequency of inbound and outbound emails while distinguishing spam, successful delivery, and encryption. While Admin Reports provide utility in incident response, they also act as a valuable tool in posture assessments.

     

    GCP – Usage Metrics & Cloud Billing Report

    Usage metrics refers to metrics, events, and metadata captured by the Cloud Monitoring service. This data can be collected from GCP, application components, on-premises environments, individual operating systems, and hybrid-cloud systems. Usage metrics can be accessed and examined through the Cloud console in “Monitoring -> Metric Explorer”.

    Cloud Billing reports provide a GCP native solution for visually representing service costs and usage over time. Accessible from within the GCP console, Cloud Billing reports allow users to apply numerous settings and filters when examining billing data through the web interface. Cloud Billing reports can be accessed through the Cloud console in “Monitoring -> Metric Explorer”.

    Regarding incident response, investigating service costs and usage can identify trends and detect anomalous activity. Depending on usage trends, Cloud Billing reports can provide a consistent source of evidence for revealing malicious activity (for example, a threat actor that created compute instances for mining activities).

     

    Logs

    Identity Management & Workspace – Log Events

    Log event data (previously called audit logs) captures identity and access events, as well as application-specific events. Depending on the subscription tier, certain application-specific logs may or may not be available.

    Log events related to either identity management or specific Workspace Apps, can be seen in the Admin Console in “Reporting -> Audit and Investigation” or via API.

    Since log event data contains multiple log types, we have highlighted a curated selection of logs that cover data critical for security engagements.

     

    Log Source

    Description

    License Requirement

    Admin Log

    Tracks actions performed in the Google Admin console

    N/A

    Groups Log

    Tracks changes to groups, group memberships, and group messages for actions taken via the Google Groups interface

    N/A

    OAuth Log

    Tracks third-party data access requests and application usage

    N/A

    SAML Log

    Tracks successful/unsuccessful sign-ins to SAML applications

    N/A

    User Log

    Tracks user events (e.g., sign-ins, password changes, 2FA setup)

    N/A

    Context Aware Access Log

    Track user access to applications (e.g., when user is denied access to an application)

    Enterprise Plus; Education Plus

    Drive Log

    View user Google Drive activity (e.g., document creation, upload, and views)

    Frontline; Business Standard or Plus; one or more Enterprise editions; Education Standard or Fundamentals; G Suite Business; Essentials

    Gmail Log

    Investigate user and admin activity related to Gmail

    Enterprise Plus or Education Plus

    Takeout Log

    View user Google Takeout activity (e.g., data export metadata)

    N/A

    Log events can be used as the main source for threat hunting activities aiming to find suspicious behavior either in Google Cloud identity management or in specific Workspace Apps. In addition, since log events document crucial time-based actions, they can be used for incident response investigation via tracking initial access into a Google Cloud ecosystem and for malicious actions done in Workspace Apps. 

    GCP – Logging Data

    Logging data captures numerous activity points in GCP. There are logs for troubleshooting, auditing control-plane modifications, capturing service specific activity, importing data from other cloud service providers, and more. To help classify the high number of logs, we have organized everything into categories derived from Google documentation.

    Logging data can be accessed via the Google Cloud console in “Logging -> Log Explorer”, GCloud, or APIs.

    Since logging data is divided into multiple categories, we have highlighted several selected logs that cover data critical for security engagements.

     

    Log Category

    Log Source

    Default Enabled

    Description

    Security

    Admin Activity Audit Log

    Yes

    Tracks API calls and other actions that modify the configuration or metadata of resources

    Data Access Audit Log

    No

    Tracks API calls that read the configuration or metadata of resources; additionally, this logs user-driven API calls that create, modify, or read user-provided resource data

    System Event Audit Log

    Yes

    Tracks non-user driven Google Cloud actions that modify the configuration of resources

    Policy Denied Audit Log

    Yes

    Tracks when a service denies access to a user or service account due to a security policy violation

    Platform

    Platform Log

    Yes

    Tracks events for service-specific Google Cloud APIs to debug and troubleshoot service issues

    VPC Flow Log

    No

    Tracks network connection metadata to and from VM instances

    GCS Usage Log

    No

    Tracks information for all web requests made to a specific bucket

    DNS Log

    No

    Tracks queries that name servers resolve for VPC networks

    HTTP/S Load Balancer Log

    No

    Tracks load balanced connections to backend services

    Firewall Rules Log

    No

    Tracks the effects of configured VPC firewall rules

    User-written

    Agent Log

    No

    Logs collected directly from GCE VM instances and written to the Cloud Logging API by one of two agents from Cloud Operations

    The Security category arguably contains the highest availability and most valuable logs for tracking the actions (i.e., API calls tracked via the “methodName” field) taken within GCP. The Admin Activity audit log records administrative changes, while the Data Access audit log records principal-driven resource access. While Admin Activity is enabled by default and retained for 400 days, Data Access must be enabled per service and is retained for 30 days. For a full list of trackable services and their respective API calls, visit the Google APIs GitHub page.

    Logging data can be used as the main source for threat hunting activities aiming to find suspicious behavior in GCP. Additionally, they can be used for incident response investigation for tracking threat actor activity.

     

    Configurations

    Identity Management – Admin Directory

    The Admin Directory provides current information on users, groups, organizational units (OUs), directory settings, and synced objects from other directories within a Google Cloud domain.

    Additionally, a comparison analysis of groups and user settings prepares the ground for developing multiple threat hunting scenarios: identifying new users with sensitive roles, identifying changes in 2SV (2-Step Verification) settings, etc.

    The Directory service is located in the Admin Console and its individual components can be found via the Directory tab.

    Directory Level

    Description

    Users

    Provides information for user status, last sign-in, MFA, connected applications, associated groups and OU, admin roles and privileges, enabled Workspace Apps, and more

    Groups

    Provides information for the list of members, access settings, if members outside organization allowed, and more

    Organizational Units

    Provides information for organizational hierarchy via root and sub-root OU’s, which are used to segment user accounts into disjointed sets for ease of management (e.g., license assignment and MFA verification) 

    Directory Settings

    Provides information for sharing and user-visibility settings

    Directory Sync

    Provides information for synced objects from external LDAP directories

    Examining the current state of the Admin Directory can help validate anomalous activity and high-risk settings (e.g., 2SV and application-specific passwords). 

    Workspace Apps – Apps Settings

    Google Workspace supplies a set of communication and collaboration applications built for organizations. All Google Workspace communication applications, such as Gmail and Google Meet, as well as collaboration applications, such as Google Docs, Sheets, Slides and Forms – have their own settings. These settings can be partially enforced by the administrator in Google Cloud but can also be set by the users. For this reason, acquiring Google Workspace Apps settings for each user can be crucial to find security threats or for investigating an incident.

    If we take Gmail as an example, Gmail mailbox settings can be used in incident response to investigate the current state of a compromised mailbox. A compromised mailbox may include suspicious forwarding rules, suspicious delegates, suspicious send-as configuration, etc. Furthermore, with the lack of logs for Gmail in most license subscriptions, examining these settings can reveal a strong indicator of threat actor activity. Gmail settings can also be used for threat hunting scenarios once compared to a previous snapshot of the same settings.

    Using the Admin Console in “Apps -> Google Workspace”, Google administrators can manage global user settings as a way of applying policy.

    However, individual user app settings (e.g., Gmail forwarding addresses) that are often more interesting, are not visible to Google administrators using the GUI. Google administrator can access this information only using API with a service account and proper delegations. 

    GCP – Inventory & Identity and Access Management (IAM) Role Bindings

    GCP supports multiple capabilities that primarily include managing cloud resources and their access.

    Google Cloud resources are organized hierarchically. The root node that represents the organization (domain) contains service resources (e.g., Compute Engine instances and Cloud Storage buckets) contained within projects, which can optionally be organized by folders. It is important to understand that every hierarchy level (organization -> folder -> project) contains logs that references activity occurring against that resource type (i.e., a folder’s logs will record activities occurring only against that folder).

    GCP’s IAM service utilizes IAM role bindings to manage the broad configuration of resource access. Through IAM role bindings and inheritance, access can be given to a member for targeted or expansive resources.

     

    FA_Figure01
    Figure 4:
    Snippet of Role Binding diagram from Google Cloud documentation

     

    With appropriate permissions, viewing the Inventory (both the resource and service data/metadata within) can be achieved through APIs (e.g., Resource Manager or Asset Inventory), GCloud, or the Google Cloud web console. Through the web console, the Inventory can be accessed via the top left corner of the page, where it is possible to select a specific hierarchy level for examination.

    FA_Figure-1_23-04

    Figure 5: Example of resource hierarchy or “Inventory” in GCP

     

    Examining GCP Inventory and IAM Role Bindings can highlight unfamiliar assets and unexpected privilege relationships.

    Conclusion

    In crafting a mental model, prioritizing evidence sources, and examining forensic artifacts, we have built a practical approach to incident response investigations in Google Cloud. Although the threat landscape is constantly evolving, it is important we maintain current knowledge by periodically reviewing Google Cloud documentation and security notices.

    Stay tuned for the final blog post and webinar "Incident Response in Google Cloud" where Sygnia will cover the release of its forensic collection tool and walk through simulated compromise scenarios!

    To learn more about Sygnia and its cybersecurity expertise, visit: www.sygnia.co

    Contributors: Wesley Guerra, Itay Angi, Oren Biderman, Shani Adir, Itay Shohat.

    This advisory and any information or recommendation contained herein has been prepared for general informational purposes and is not intended to be used as a substitute for professional consultation on facts and circumstances specific to any entity. While we have made attempts to ensure the information contained herein has been obtained from reliable sources and to perform rigorous analysis, this advisory is based on initial rapid study, and needs to be treated accordingly. Sygnia is not responsible for any errors or omissions, or for the results obtained from the use of this Advisory. This Advisory is provided on an as-is basis, and without warranties of any kind.

     

    Back to Resources