Harvesting Overview

Overview Information on InsightCloudSec Harvesting (aka Resource Collection) Capabilities

Summary

Harvesting is the term we use to describe the process of data collection from a selected cloud service provider (CSP) within InsightCloudSec. Harvesting is performed on a standard polling schedule that can be customized by service, region, account, or cloud badge. This approach allows the InsightCloudSec platform to provide flexibility for customers around data collection based on their organization’s requirements.

For example, you can lower the harvesting cadence in regions where you do not have infrastructure deployed to reduce redundant API calls, but still maintain some harvesting to monitor for unauthorized usage in those regions. Administrators have the ability to fine-tune configurations on harvesting strategies for each resource type, either per region or globally, for any cloud provider.

InsightCloudSec - Harvesting StrategiesInsightCloudSec - Harvesting Strategies

InsightCloudSec - Harvesting Strategies

General Harvesting - FAQ

Where can I view the harvesting status of a particular resource?

  • Each resource includes a details view with a "Last Full Harvest" field. Navigate to the Resources page, select your resource type, click on the actions menu and select "View Details" to open the detailed resource view.

    • Note that this field may be blank because a resource was not recently harvested or the Redis cache (AWS-Only) was updated and the status is no longer stored in the database.

How can I manually trigger job harvesting?

  • Select an individual cloud account under "Cloud --> Clouds --> Listing", and select the "Harvest Info" tab. Scroll to the right to select "Enqueue Now" to trigger harvesting for a single job, or select multiple jobs under the Cloud Account and then click "Enqueue Selected".
Manual Job HarvestingManual Job Harvesting

Manual Job Harvesting

Event-Driven Harvesting (EDH)

In addition to our basic harvesting, InsightCloudSec offers advanced harvesting through our Event-Driven Harvesting (EDH). This capability is used to augment standard polling-based harvesting. For AWS EDH pulls data from AWS CloudWatch Events and AWS CloudTrail into a central Eventbus, and for Azure it uses EventGrid for InsightCloudSec's consumption.

By using events to initiate resource-specific harvesting, EDH improves the timeliness of resource updates and provides opportunities for rapid notification and/or remediation. The harvested events also enrich the data by identifying how resources enter noncompliant states. This detail enables auditing capabilities that function at scale.

Review AWS Event-Driven Harvesting for more information on EDH with AWS.

Review Azure Event-Driven Harvesting for more information on EDH with Azure.

Event-Driven HarvestingEvent-Driven Harvesting

Event-Driven Harvesting

Event-Driven Harvesting - FAQ

How long are EDH Events Retained?

  • EDH Events are retained for five days.

What happens if AWS CloudWatch/Azure sends all events to InsightCloudSec?

  • InsightCloudSec uses rules to process specific events for supported services. Other events are dropped.

How do rules get updated?

  • This varies depending on your deployment and configuration. InsightCloudSec can manage these rules automatically for you, or you may have elected to do so yourself in a manual way.

Can I manage the set up of EDH on my own and not give InsightCloudSec “write” permissions?

  • Yes for AWS this requires support to assist with a custom configuration.
  • For Azure this can be configured within the Azure Console

Reach out to us through the Customer Support Portal for assistance in either scenario.

Where should I look to start troubleshooting issues with messages not showing up?

  • SQS (AWS)
  • Cloud Collector Logs (Azure)
  • Docker logs
  • In the InsightCloudSec platform, as some logs are visible in the EDH section and in the admin logs section

When enabled EDH (for Azure) how long does auto-detection take for a new EDH Setup?
For Azure EDH the consumer should be detected on the first run of the collector and any producers will be detected upon the collector receiving an event.

Using Resource MetaData from EDH

Currently only supported for AWS

Beginning with 21.3, resource Event Driven Harvesting (EDH) metadata was expanded to allow the attachment of resources harvested using EDH to include the IP address of the resource’s creator.

Earlier, we introduced the ability to autotag provisioned resources with a “divvy.creator” tag, which is useful for tracking resource ownership and maintaining accountability. Now, we can also autotag provisioned resources with a “divvy.creator_ip” tag, which captures the IP address of the user who created the resource.

As part of these updates we added a new filter, {{Resource Provisioned From Unauthorized Network (AWS)}}, that compares resource “creator_ip” addresses with authorized IPv4 networks to identify resources provisioned from outside of those networks.

To make the filter more useful, we recommend first creating a data collection for the filter called “Authorized Networks”. The data collection should have internal approved networks supplemented with AWS networks to surface true concerns.

Next, we recommend creating a Bot that runs using the “Resource Created” hookpoint. The Bot can send notifications, take corrective actions, or both.

📘

DivvyCloud vs. InsightCloudSec

Note that some images, database items, and backend components may still reference DivvyCloud rather than InsightCloudSec, these capabilities function the same regardless of naming.


Did this page help you?