Incident Response in Google Cloud: Forensic Artifacts

Key Takeaways

• Forensic data across Google Cloud can logically be organized into three categories: Identity Management, Google Workspace Apps, and Google Cloud Platform (GCP). Each category can be further broken down into four subcategories: Configurations, Logs, Reports, and Alerts.
• During triage, prioritize the following evidence sources when performing incident response against Google Workspace:
  
  • Alert Center alerts > Admin reports > Identity logs > Application logs > Application data
• During triage, prioritize the following evidence sources when performing incident response against Google Cloud Platform:
  • Alert Center alerts > Identity logs > Security and Platform logs > Service and Resource data

Introduction

In our previous blog, we provided a foundational view on identity management in Google Cloud, which helped in understanding that evidence sources may vary depending on whether the cloud environment consists of GCP, Google Workspace, or Google Cloud Identity. We also discussed how these larger components are interrelated and from an incident response perspective, have the capacity to detrimentally affect each other. To build upon this foundation, this article will examine forensic artifacts available and provide recommendations for triage and prioritization.

Creating a Mental Model: Forensic Artifacts in Google Cloud

Mental models serve to simplify complex scenarios and create an approach to reasoning through them. Because there are many artifacts spread throughout Google Cloud across a multitude of services and locations, we will categorize everything for ease of tracking. While there are limited or paid-tier services (e.g., Security Investigation Tool in Google Workspace and Security Command Center in GCP) that perform security-related functions useful during incident response, we will focus on the most widely available evidence sources.

A basic tenant of incident response is to understand in all data sources where potential anomalous or malicious activity can occur. For this reason, Google Cloud data sources have been broken down into three categories: identity management for the overall Google Cloud, applications specific to Workspace, and resources and services within GCP. Each of these categories are broken down further into subcategories:

• Configuration: service, application, and access settings and configurations.
• Logs: track administrative actions and access across Google Cloud resources.
• Reports: statistical information, presented in pre-built graphs and tables.
• Alerts: provides Google pre-configured and custom alerting.

Google Cloud forensic data can be accessed through the Admin Console or via APIs for Google Cloud identity management and Google Workspace, and through Google Cloud console, Google Cloud CLI (GCloud), or via APIs for GCP.

We will first begin by recommending evidence prioritization via triage, and subsequently examine the data sources to better understand their role in threat hunting and incident response.

Triage in Google Cloud

Triage is the process of assigning priority to sources of evidence and affected assets during a security incident based on efficacy and risk. The ability to quickly identify, prioritize, and resolve security events is critical when managing or responding to incidents within a Google Cloud environment.

In the sections below, we have designated artifact priority based on the likelihood of availability and utility of data captured. The sections provide an example for triage in Workspace and GCP separately, while considering identity-based evidence across both. The steps of the triage may vary based on the incident details, priority, and resources.

Triage in Workspace involves its managed identities and their interactions with the productivity applications. With an examination of the Alert Center in the Admin console, we can quickly identify anomalous activity associated with specific identities or applications.
Once a preliminary review of alerts has been completed, Admin reports can provide a high-level overview of productivity application use and potential abuse. For example, we can identify potential phishing campaigns launched from Workspace via Gmail reports, specifically through analysis of outbound email delivery spikes. The identity and application logs can then be used to pivot from compromised accounts to discover additional events and indicators of compromise (IOCs). Once the review of alerts, reports, and logs has been concluded, a deeper dive into specific application data can begin (e.g., gathering email data) if necessary.

Triage in Google Cloud Platform

Triage in GCP involves the domain’s identities and their interactions with its services and resources. As the Alert Center is available in the Admin Console by default, its alerts can provide easily identifiable IOCs to pivot from. After reviewing alerts, the identity logs (User log, Admin log, SAML log, etc.) will help to pinpoint any abnormal access into the Google Cloud domain. Once the affected identities have been scoped, identifying actions taken within GCP itself is observable in the Security and Platform logs. More specifically, begin by examining the default enabled Admin Activity audit log for GCP resource modifications and the Data Access audit log (if available) for user-driven resource access. Once these initial artifacts have been exhausted and a better grasp of the security incident has been achieved, the investigation can be conducted effectively by targeting precise services or resources. 

After exploring the triage methodology, we will now cover each described data source to understand what it consists of and its forensic value for both incident response and threat hunting.

Alerts

Identity Management & Workspace – Alert Center Alerts

Alert Center alerts provide pre-configured and custom alerting on potential issues within a Google Cloud domain, either by Google Cloud’s identity management or Workspace Apps. Pre-configured alerts are enabled by default and should be one of the first sources of evidence examined when dealing with a security incident. These alerts capture abnormal user, administrative, and device events. Many security incidents that result in large-scale compromise can often be prevented or mitigated by reviewing security alerts as they occur.

The Alert Center is located in the Admin Console in “Security tab -> Alert center” and retains data for approximately ten years. Although a curated selection is described below, visit Google documentation for a full list of alerts.

Name	Description
General alerts	Security and privacy issues, government-backed attacks
User alerts	Suspicious login, user granted admin privilege, user suspended, password change, suspicious programmatic login
Administrative alerts	Super admin password reset, primary admin changed, SSO profile added, domain data export initiated
Gmail alerts	Employee spoofing, malware detected post-delivery, suspicious messages, user-reported phishing
Custom alerts	Manually configured activity, reporting, and DLP alerts

Reports

Identity Management & Workspace – Admin Reports

Admin Reports summarize user security-oriented settings, application activity, and billing costs in a statistics form. Admin Reports consist of data from Google Cloud’s identity management and Workspace Apps. The statistics report, which are either presented as graphs or as tables provide easily digestible output that can be utilized as a quick method to identify anomalous activity.

Admin Reports are found in the Admin Console in “Reporting -> Reports” and are retained for six months. Although there are more reports available, we can focus on the ones that can be more easily utilized for security purposes.

Category	Name	Description
Highlights	Highlights	Captures Workspace App use, user status, storage quota, document visibility, and security metrics
Apps Reports	Accounts	Captures organizational security-related setting metrics
	Aggregate Reports	Captures metrics across all Google Apps
	Drive	Captures active user, file, and share metrics
	Gmail	Captures email metrics (e.g., emails sent/received)
User Reports	Apps Usage	Captures user-based Google Workspace Apps usage metrics (e.g., last Gmail access time through web or legacy protocols, storage quotas, file creation); Captured Workspace Apps depend on Google Workspace license
	Security	Captures security-related setting status per user
	Apps Usage	Captures user-based Google Workspace Apps usage metrics (e.g., last Gmail access time through web or legacy protocols, storage quotas, file creation); Captured Workspace Apps depend on Google Workspace license

Although many of the metrics and trends covered by Admin Reports do not provide practical evidence for the incident response process – Gmail report, user-based Apps Usage report, and cost report can be helpful in identifying irregular application usage. For example, the Gmail app report will track the frequency of inbound and outbound emails while distinguishing spam, successful delivery, and encryption. While Admin Reports provide utility in incident response, they also act as a valuable tool in posture assessments.

GCP – Usage Metrics & Cloud Billing Report

Usage metrics refers to metrics, events, and metadata captured by the Cloud Monitoring service. This data can be collected from GCP, application components, on-premises environments, individual operating systems, and hybrid-cloud systems. Usage metrics can be accessed and examined through the Cloud console in “Monitoring -> Metric Explorer”.

Cloud Billing reports provide a GCP native solution for visually representing service costs and usage over time. Accessible from within the GCP console, Cloud Billing reports allow users to apply numerous settings and filters when examining billing data through the web interface. Cloud Billing reports can be accessed through the Cloud console in “Monitoring -> Metric Explorer”.

Regarding incident response, investigating service costs and usage can identify trends and detect anomalous activity. Depending on usage trends, Cloud Billing reports can provide a consistent source of evidence for revealing malicious activity (for example, a threat actor that created compute instances for mining activities).

LOGS

Identity Management & Workspace – Log Events

Log event data (previously called audit logs) captures identity and access events, as well as application-specific events. Depending on the subscription tier, certain application-specific logs may or may not be available.

Log events related to either identity management or specific Workspace Apps, can be seen in the Admin Console in “Reporting -> Audit and Investigation” or via API.

Since log event data contains multiple log types, we have highlighted a curated selection of logs that cover data critical for security engagements.

Log Source	Description	License Requirement
Admin Log	Tracks actions performed in the Google Admin console	N/A
Groups Log	Tracks changes to groups, group memberships, and group messages for actions taken via the Google Groups interface	N/A
OAuth Log	Tracks third-party data access requests and application usage	N/A
SAML Log	Tracks successful/unsuccessful sign-ins to SAML applications	N/A
User Log	Tracks user events (e.g., sign-ins, password changes, 2FA setup)	N/A
Context Aware Access Log	Track user access to applications (e.g., when user is denied access to an application)	Enterprise Plus; Education Plus
Drive Log	View user Google Drive activity (e.g., document creation, upload, and views)	Frontline; Business Standard or Plus; one or more Enterprise editions; Education Standard or Fundamentals; G Suite Business; Essentials
Gmail Log	Investigate user and admin activity related to Gmail	Enterprise Plus or Education Plus
Takeout Log	View user Google Takeout activity (e.g., data export metadata)	N/A

Log events can be used as the main source for threat hunting activities aiming to find suspicious behavior either in Google Cloud identity management or in specific Workspace Apps. In addition, since log events document crucial time-based actions, they can be used for incident response investigation via tracking initial access into a Google Cloud ecosystem and for malicious actions done in Workspace Apps. 

GCP – Logging Data

Logging data captures numerous activity points in GCP. There are logs for troubleshooting, auditing control-plane modifications, capturing service specific activity, importing data from other cloud service providers, and more. To help classify the high number of logs, we have organized everything into categories derived from Google documentation.

Logging data can be accessed via the Google Cloud console in “Logging -> Log Explorer”, GCloud, or APIs.

Since logging data is divided into multiple categories, we have highlighted several selected logs that cover data critical for security engagements.

Log Category	Log Source	Default Enabled	Description
Security	Admin Activity Audit Log	Yes	Tracks API calls and other actions that modify the configuration or metadata of resources
	Data Access Audit Log	No	Tracks API calls that read the configuration or metadata of resources; additionally, this logs user-driven API calls that create, modify, or read user-provided resource data
	System Event Audit Log	Yes	Tracks non-user driven Google Cloud actions that modify the configuration of resources
	Policy Denied Audit Log	Yes	Tracks when a service denies access to a user or service account due to a security policy violation
Platform	Platform Log	Yes	Tracks events for service-specific Google Cloud APIs to debug and troubleshoot service issues
	VPC Flow Log	No	Tracks network connection metadata to and from VM instances
	GCS Usage Log	No	Tracks information for all web requests made to a specific bucket
	DNS Log	No	Tracks queries that name servers resolve for VPC networks
	HTTP/S Load Balancer Log	No	Tracks load balanced connections to backend services
	Firewall Rules Log	No	Tracks the effects of configured VPC firewall rules
User-written	Agent Log	No	Logs collected directly from GCE VM instances and written to the Cloud Logging API by one of two agents from Cloud Operations

The Security category arguably contains the highest availability and most valuable logs for tracking the actions (i.e., API calls tracked via the “methodName” field) taken within GCP. The Admin Activity audit log records administrative changes, while the Data Access audit log records principal-driven resource access. While Admin Activity is enabled by default and retained for 400 days, Data Access must be enabled per service and is retained for 30 days. For a full list of trackable services and their respective API calls, visit the Google APIs GitHub page.

Logging data can be used as the main source for threat hunting activities aiming to find suspicious behavior in GCP. Additionally, they can be used for incident response investigation for tracking threat actor activity.

Configuration

Identity Management – Admin Directory

The Admin Directory provides current information on users, groups, organizational units (OUs), directory settings, and synced objects from other directories within a Google Cloud domain.

Additionally, a comparison analysis of groups and user settings prepares the ground for developing multiple threat hunting scenarios: identifying new users with sensitive roles, identifying changes in 2SV (2-Step Verification) settings, etc.

The Directory service is located in the Admin Console and its individual components can be found via the Directory tab.

Directory Level	Description
Users	Provides information for user status, last sign-in, MFA, connected applications, associated groups and OU, admin roles and privileges, enabled Workspace Apps, and more
Groups	Provides information for the list of members, access settings, if members outside organization allowed, and more
Organizational Units	Provides information for organizational hierarchy via root and sub-root OU’s, which are used to segment user accounts into disjointed sets for ease of management (e.g., license assignment and MFA verification)
Directory Settings	Provides information for sharing and user-visibility settings
Directory Sync	Provides information for synced objects from external LDAP directories

Examining the current state of the Admin Directory can help validate anomalous activity and high-risk settings (e.g., 2SV and application-specific passwords). 

Workspace Apps – Apps Settings

Google Workspace supplies a set of communication and collaboration applications built for organizations. All Google Workspace communication applications, such as Gmail and Google Meet, as well as collaboration applications, such as Google Docs, Sheets, Slides and Forms – have their own settings. These settings can be partially enforced by the administrator in Google Cloud but can also be set by the users. For this reason, acquiring Google Workspace Apps settings for each user can be crucial to find security threats or for investigating an incident.

If we take Gmail as an example, Gmail mailbox settings can be used in incident response to investigate the current state of a compromised mailbox. A compromised mailbox may include suspicious forwarding rules, suspicious delegates, suspicious send-as configuration, etc. Furthermore, with the lack of logs for Gmail in most license subscriptions, examining these settings can reveal a strong indicator of threat actor activity. Gmail settings can also be used for threat hunting scenarios once compared to a previous snapshot of the same settings.

Using the Admin Console in “Apps -> Google Workspace”, Google administrators can manage global user settings as a way of applying policy.

However, individual user app settings (e.g., Gmail forwarding addresses) that are often more interesting, are not visible to Google administrators using the GUI. Google administrator can access this information only using API with a service account and proper delegations. 

GCP – Inventory & Identity and Access Management (IAM) Role Bindings

GCP supports multiple capabilities that primarily include managing cloud resources and their access.

Google Cloud resources are organized hierarchically. The root node that represents the organization (domain) contains service resources (e.g., Compute Engine instances and Cloud Storage buckets) contained within projects, which can optionally be organized by folders. It is important to understand that every hierarchy level (organization -> folder -> project) contains logs that references activity occurring against that resource type (i.e., a folder’s logs will record activities occurring only against that folder).

GCP’s IAM service utilizes IAM role bindings to manage the broad configuration of resource access. Through IAM role bindings and inheritance, access can be given to a member for targeted or expansive resources.

With appropriate permissions, viewing the Inventory (both the resource and service data/metadata within) can be achieved through APIs (e.g., Resource Manager or Asset Inventory), GCloud, or the Google Cloud web console. Through the web console, the Inventory can be accessed via the top left corner of the page, where it is possible to select a specific hierarchy level for examination.

Examining GCP Inventory and IAM Role Bindings can highlight unfamiliar assets and unexpected privilege relationships.

Conclusion

In crafting a mental model, prioritizing evidence sources, and examining forensic artifacts, we have built a practical approach to incident response investigations in Google Cloud. Although the threat landscape is constantly evolving, it is important we maintain current knowledge by periodically reviewing Google Cloud documentation and security notices.

Stay tuned for the final blog post and webinar “Incident Response in Google Cloud” where Sygnia will cover the release of its forensic collection tool and walk through simulated compromise scenarios!

Google Cloud forensics blog series

• Incident Response in Google Cloud: Foundations
• Incident Response in Google Cloud: Forensic Artifacts
• Announcing ‘Cirrus’ – New Opensource Tool to Combat Google Cloud Incident Response Challenges
• Proof of Concept: Overcoming Google Cloud Incident Response Issues with ‘Cirrus’

This advisory and any information or recommendation contained herein has been prepared for general informational purposes and is not intended to be used as a substitute for professional consultation on facts and circumstances specific to any entity. While we have made attempts to ensure the information contained herein has been obtained from reliable sources and to perform rigorous analysis, this advisory is based on initial rapid study, and needs to be treated accordingly. Sygnia is not responsible for any errors or omissions, or for the results obtained from the use of this Advisory. This Advisory is provided on an as-is basis, and without warranties of any kind.