EDH - GCP Setup
Step-by-step Setup Instructions for Configuration of GCP Event-Driven Harvesting
The instructions below outline the steps required to complete the setup of GCP Event-driven Harvesting (EDH) in InsightCloudSec using the GCP Console and Cloud Shell.
For InsightCloudSec to harvest data when triggered by events or changes in your GCP environment, you'll need to ensure that a service account is properly able to consume events, create a Pub/Sub topic, and create a Cloud Asset Inventory (CAI) feed. Review GCP's documentation for more information on these concepts. If setup properly, the following will occur:
1. GCP resource update events are captured by a CAI feed. Note: feeds support filtering per resource type.
2. The CAI feed routes events to a Pub/Sub topic and the topic ensures the events arrive in a paired Pub/Sub subscription.
3. InsightCloudSec monitors the subscription for new events and enqueues harvesters for the relevant resources.
Before you configure anything in your GCP environment, you'll need the following:
- A functioning InsightCloudSec platform with the appropriate Admin (Org or Domain) permissions
- A GCP Organization added to InsightCloudSec (i.e., an existing GCP Service Account associated with InsightCloudSec)
- Review Organizations (GCP) for more information
- A basic understanding of the relevant GCP services (Cloud Asset Inventory, Pub/Sub, Service Account)
- Appropriate GCP administrative permissions
- The ability to create Pub/Sub topics within a project
- Note: The Cloud Asset Owner role has all the requisite Organization IAM permissions.
- At a minimum, the Cloud Asset Inventory API enabled within GCP
- Note: Some supported resources for EDH may require additional APIs; cross-reference the General GCP Recommended APIs with the list of Supported GCP EDH Resources/Asset Types
If you would rather only enable EDH for a single project instead of an entire GCP organization, reach out to us through the Customer Support Portal.
Step 1: Create a Pub/Sub Topic and Subscription
First, you'll have to create a matching Pub/Sub topic and subscription that will be used to publish and consume events. This topic and subscription pair needs to be created in the same project that contains the Service Account you created during the GCP Organization setup.
1. Within the Google Cloud Console, open the project containing the service account that is currently integrated with InsightCloudSec.
- Navigate to "Pub/Sub".
- Note: If you've never used Pub/Sub for the project before, it may take a second to enable the API.
2. Click "Create Topic".
- For the Topic ID, type
- Ensure "Add a default subscription" is selected. *Note: Once the topic is created, this option will ensure a matching subscription is also created with an ID of
- Click "Create Topic".
3. After the topic is successfully created, copy the Subscription ID and keep it on hand for later.
Step 2: Update Service Account Permissions
Next, you must ensure the Service Account that will be used to consume events has the
pubsub.subscriptions.consume permission to grab the events from the Pub/Sub topic. The easiest and best way to assure this is to assign the Service Account the Pub/Sub Subscriber role directly from the subscription itself. Note: The same Service Account created during the GCP Organization integration process should be used for the GCP EDH setup.
Before you can update the Service Account's permissions, you will need the following on hand:
- The full Service Account name (created in Organizations (GCP))
1. Within the Google Cloud Console, from the "Pub/Sub" page, navigate to "Subscriptions".
2. Select the subscription you just created in the previous section and expand the info panel (top right corner).
- Click "Add Principal".
3. On the Grant Access panel, update the service account with the proper permissions.
- Provide the service account name that's currently used by InsightCloudSec. Note: It's easiest if you paste the full name into the field because GCP may not be able to auto-complete the value if you only type in a partial name.
- From the roles drop-down menu, filter for and select "Pub/Sub Subscriber".
- Click "Save".
Step 3: Create a CAI Feed
Before you can finish up the configuration within the InsightCloudSec UI, you'll need to create a CAI feed to send events to the topic. This section will assist in setting up a feed within the project that you created a topic/subscription in and is integrated with InsightCloudSec. Currently, feeds cannot be created via the GCP Console UI, so it will need to be created via the Cloud Shell, which is conveniently located in the GCP Console.
Before you can successfully create a CAI feed, you will need the following on hand:
- The project ID containing the service account & Pub/Sub topic and subscription (the Project ID can be found using these instructions)
- The Pub/Sub topic name (created in step 1)
- The organization ID housing the project you're setting up for EDH (the Organization ID can be found using these instructions)
- A comma-delimited list of asset types (found on EDH - Supported Resources)
1. Within the Google Cloud Console, click the "Activate Cloud Shell" icon in the top right corner.
2. Copy the following code snippet and paste it into the Cloud Shell window, ensuring you replace:
<project_id>with the appropriate GCP Project ID
<topic_name>with the appropriate Pub/Sub topic name
<organization_id>with the appropriate GCP organization ID
- Note: The full list of asset types has been included in the example command below
gcloud asset feeds create r7-gcp-edh-resource-feed \ --pubsub-topic=projects/<project_id>/topics/<topic_name> \ --organization=<organization_id> \ --content-type=resource \ --asset-types=compute.googleapis.com/Autoscaler,bigtableadmin.googleapis.com/Instance,compute.googleapis.com/SslCertificate,file.googleapis.com/Instance,cloudfunctions.googleapis.com/CloudFunction,run.googleapis.com/Service,compute.googleapis.com/TargetVpnGateway,compute.googleapis.com/VpnTunnel,compute.googleapis.com/Image,container.googleapis.com/Cluster,compute.googleapis.com/Network,iam.googleapis.com/Role,secretmanager.googleapis.com/Secret,compute.googleapis.com/SecurityPolicy,iam.googleapis.com/ServiceAccountKey,compute.googleapis.com/Snapshot,compute.googleapis.com/Subnetwork,pubsub.googleapis.com/Subscription,pubsub.googleapis.com/Topic
3. Press "Enter". Note: If you have not used the CAI API before, you will be prompted to enable it and retry the command. Confirm this action.
3. Optionally, run the following command in the Cloud Shell window to verify the feed was created successfully, ensuring you replace
<project_id> with the appropriate GCP Project ID:
gcloud asset feeds list --project <project_id>
Step 4: Configure InsightCloudSec
Now that GCP has be configured for EDH, it's time to configure InsightCloudSec.
Before you can successfully configure EDH within InsightCloudSec, you will need the following on hand:
- The fully-qualified subscription ID, which contains the project ID, e.g.,
projects/<project_id>/subscriptions/<subscription_id>(created in step 1)
1. Login to your InsightCloudSec platform and click "Clouds" in the left-hand navigation menu.
- Click "Consumers".
- Click "EDH Configuration".
- From the drop-down menu, click "GCP Pub/Sub Consumer".
2. Update the configuration for the necessary information.
- Select the GCP Project that contains the service account & Pub/Sub topic and subscription.
- Provide the fully-qualified project subscription ID.
- Click "Configure".
Post Setup Information
Congratulations on setting up Event Driven Harvesting for your GCP Organizations within InsightCloudSec. Below you'll find some important links about EDH in general.
Updated 5 months ago