InsightCloudSec Docs

Welcome to the InsightCloudSec Docs!

InsightCloudSec by Rapid7 (formerly DivvyCloud) is a Cloud-Native Security Platform that provides real-time analysis and automated remediation for continuous security and compliance for your multi-cloud environment.

For questions reach out to us through [email protected].

Take Me to the Docs!    Release Notes

Harvesting Overview

Overview Information on InsightCloudSec Harvesting (aka Resource Collection) Capabilities


Harvesting is the term we use to describe the process of data collection from a selected cloud service provider (CSP) within InsightCloudSec. Harvesting is performed on a standard polling schedule that can be customized by service, region, account, or cloud badge. This approach allows the InsightCloudSec platform to provide flexibility for customers around data collection based on their organization’s requirements.

For example, you can lower the harvesting cadence in regions where you do not have infrastructure deployed to reduce redundant API calls, but still maintain some harvesting to monitor for unauthorized usage in those regions. Administrators have the ability to fine-tune configurations on harvesting strategies for each resource type, either per region or globally, for any cloud provider.

InsightCloudSec - Harvesting StrategiesInsightCloudSec - Harvesting Strategies

InsightCloudSec - Harvesting Strategies

General Harvesting - FAQ

Where can I view the harvesting status of a particular resource?

  • Each resource includes a details view with a "Last Full Harvest" field. Navigate to the Resources page, select your resource type, click on the actions menu and select "View Details" to open the detailed resource view.

    • Note that this field may be blank because a resource was not recently harvested or the Redis cache was updated and the status is no longer stored in the database.

How can I manually trigger job harvesting?

  • Select an individual cloud account under "Cloud --> Clouds --> Listing", and select the "Harvest Info" tab. Scroll to the right to select "Enqueue Now" to trigger harvesting for a single job, or select multiple jobs under the Cloud Account and then click "Enqueue Selected".
Manual Job HarvestingManual Job Harvesting

Manual Job Harvesting

Event-Driven Harvesting (EDH)

In addition to our basic harvesting, InsightCloudSec offers advanced harvesting through our Event-Driven Harvesting (EDH). This capability is used to augment standard polling-based harvesting and pulls data from AWS CloudWatch Events and AWS CloudTrail into a central Eventbus for InsightCloudSec's consumption. (While this capability is currently only available for AWS, EDH is currently in development for Microsoft Azure).

By using events to initiate resource-specific harvesting, EDH improves the timeliness of resource updates and provides opportunities for rapid notification and/or remediation. The harvested events also enrich the data by identifying how resources enter noncompliant states. This detail enables auditing capabilities that function at scale.

Review AWS Event-Driven Harvesting for more information.

Event-Driven HarvestingEvent-Driven Harvesting

Event-Driven Harvesting

Event-Driven Harvesting - FAQ

How long are EDH Events Retained?

  • EDH Events are retained for five days.

What happens if AWS CloudWatch sends all events to InsightCloudSec?

  • InsightCloudSec uses rules to process specific events for supported services. Other events are dropped.

How do rules get updated?

  • This varies depending on your deployment and configuration. InsightCloudSec can manage these rules automatically for you, or you may have elected to do so yourself in a manual way.

Can I manage the set up of EDH on my own and not give InsightCloudSec “write” permissions?

  • Yes, but this requires support to assist with a custom configuration. Reach out to us through [email protected].

Where should I look to start troubleshooting issues with messages not showing up?

  • SQS
  • Docker logs
  • In the InsightCloudSec platform, as some logs are visible in the EDH section and in the admin logs section

Using Resource MetaData from EDH

Beginning with 21.3, resource Event Driven Harvesting (EDH) metadata was expanded to allow the attachment of resources harvested using EDH to include the IP address of the resource’s creator.

Earlier, we introduced the ability to autotag provisioned resources with a “divvy.creator” tag, which is useful for tracking resource ownership and maintaining accountability. Now, we can also autotag provisioned resources with a “divvy.creator_ip” tag, which captures the IP address of the user who created the resource.

As part of these updates we added a new filter, {{Resource Provisioned From Unauthorized Network (AWS)}}, that compares resource “creator_ip” addresses with authorized IPv4 networks to identify resources provisioned from outside of those networks.

To make the filter more useful, we recommend first creating a data collection for the filter called “Authorized Networks”. The data collection should have internal approved networks supplemented with AWS networks to surface true concerns.
Next, we recommend creating a Bot that runs using the “Resource Created” hookpoint. The Bot can send notifications, take corrective actions, or both.

Updated about a month ago

Harvesting Overview

Overview Information on InsightCloudSec Harvesting (aka Resource Collection) Capabilities

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.