InsightCloudSec Docs

Welcome to the InsightCloudSec Docs!

InsightCloudSec by Rapid7 (formerly DivvyCloud) is a Cloud-Native Security Platform that provides real-time analysis and automated remediation for continuous security and compliance for your multi-cloud environment.

For questions reach out to us through [email protected].

Take Me to the Docs!    Release Notes

Working with Bots (Best Practices & Examples)

Examples of Formatting and Implementation of Bots to Understand InsightCloudSec Automation

Overview

As a best practice, InsightCloudSec recommends creating Bots from Insights. The benefits of creating a Bot from an Insight are numerous but revolve around two key advantages: visibility and updates.

Bots that are created from a source Insight are linked to that Insight; whether you choose to build the Bot from an existing InsightCloudSec Insight or a custom Insight, you will always be able to see the relationship between the two. A Bot created from an Insight:

  • Enables you to keep track of how many Bots are tied to a specific Insight. Bots built ad-hoc are only traceable via individual Filters (at last count, InsightCloudSec has over 1300).

  • Include Exemptions (which are directly linked to Insights) in your Bot's scope or behavior. Bots built ad-hoc will not support exemptions.

  • Benefit from updates around Insights or filters that may apply to the associated Bot (e.g., a filter includes an improvement to provide better results, which your Bot can then take action around).

    • For example, if any updates are made to the Insight (e.g., to add/modify/remove a filter, or changes to exemptions) the Bot will inherit these changes through the linked Insight.
Number of Bots associated with a Specific InsightNumber of Bots associated with a Specific Insight

Number of Bots associated with a Specific Insight

🚧

Caution: Before Getting Started

Since every environment is different, we can offer no guarantees on the results, performance, or outcomes of any Bot example provided here. This includes performance specific to any Cloud Service Provider (e.g., some example Bots may perform well for GCP or Azure but not AWS and vice versa).

If you have access to a testing environment, in general we'd recommend verifying that your Bot performs as expected there before deploying to production. In addition, if you have general questions or concerns we are always happy to assist! Contact us at [email protected]

Exploring Bot Implementation

One way to familiarize yourself with Bots and learn more about using InsightCloudSec's automation through BotFactory is by reviewing and understanding both Resources and Insights.

Starting from Resources

Beginning from the Resources section of InsightCloudSec, you can focus on a specific resource type and examine all of the filters associated with that resource. Building any Insight involves selecting from hundreds of filters to help identify resources based on characteristics and filters.

For example, if you want to view all instances not in the Continental United States, you could view instances on the Resources page, apply and then appropriately configure the “Resource not in Region” filter.

  • This scoped resource view can be saved as an Insight.
  • The newly saved Custom Insight can be configured with additional parameters and used as the basis for creating a Bot.

A Bot created from this example Insight could be configured with actions to do any or all of the following:

  • Notify the Org Admin when one of these instances is created
  • Send the instance owner a reminder of corporate policy on resource location
  • Pause or delete the instance

Starting with Insights

Another method of Bot creation takes advantage of an existing Insight. For many users the best approach to creating a custom Bot begins by using an existing InsightCloudSec Insight or even a custom Insight that is relevant to your needs.

In either scenario, a "one-click" Bot creation will build a Bot template focused on the details defined by that specific Insight.

For example, if you want to use an existing Insight, you can simply locate the Insight by name and click on the actions menu to launch the "Create Bot" process.

If you want to use an InsightCloudSec Insight as a starting point to customize and then build automation through a Bot, the best way is to locate the Insight and open the Impacted Resources column (this provides a scoped Resources view, which you can then edit and save as a Custom Insight).

Understanding Example Bots

This section includes a number of Bot examples and their underlying architecture. To be clear, this list demonstrates just a few of the possibilities for deploying Bots.

  • We want you to explore examples through their components and behaviors to provide a closer look at the mechanics of a Bot.
  • By reviewing these examples we hope you can get a better sense of some of the possibilities for how to build, deploy, and customize Bots and automation in your environment.

🚧

Important Notes - Recommended Approach

For each example provided below we recommend that you:
- Use Filters defined in the Bot to create a Custom Insight
- Use this Custom Insight as a starting point (via the actions menu) to build the Bot

If you choose to build any of the Bots outlined below (or any Bot for that matter) without an Insight as your starting point you will have to manually maintain the Bot including any additions/modifications/updates to the Filters or Scope.

In addition, Bots that are not created from an Insight will not include Exemptions because those are defined at the Insight level.

Permissions & Entitlements - In order to create Bots, basic users will require "Editor" or "Admin" rights under Permissions Entitlements. Reach out to your administrator or to [email protected] to address permission or entitlement issues.

Understanding Example Bot Formatting

The table below explains the format for the sample bots that follow.

Bot Configuration Component

Configuration Details

Scope

Defines the scope of the Bot, typically the specified Resource Type (e.g., Access List, Instances, Databases, etc.) AND the cloud accounts that the bot will take action in.

In general we recommend using the Badges functionality to manage scope. Details are available in the Badges documentation.

Filters

Filters are applied to the Bot (or if you are following the recommended approach, you will select your desired filter(s) and save these as your Custom Insight)

Action (1)

Defines the first thing that your Bot "does", e.g., send a Slack message, send an email, etc.

Jinja2 Template

Jinja2 Templating to craft your message. Check out Jinja2 for more information about using this feature.

Action (2)

Defines the additional thing that your Bot "does", e.g., stop an Instance, disable a user, etc.*

Permissions

Permissions are included where applicable but vary based on the behaviors of the Bot. Most permissions for users revolve around access to cloud resources.

  • For example, if you only have "View" permissions, your Bot (which will inherit permissions of the creator) will not be able to take any lifecycle actions (e.g., start, stop, edit, etc.) on resources.
  • If you have "Modify" permissions, some additional actions are available.
  • In order to "Delete", specific delete permissions are required.

In general, groups and roles are how users interact with cloud resources. Refer to User, Groups, and Roles (Administration) for details on permissions around users.

Entitlements manage feature usage within the InsightCloudSec platform. Details on Entitlements are available under our documentation on Identity Management.

Example Bots

Refer to the details on each individual Bot below. These are recommendations but can always be adjusted to suit your environment or requirements. If you need help with the specifics or other configuration details, reach out to us through [email protected].

Security Rules Exposing Public Access to Vulnerable Services

This Bot identifies access lists with (non web) ports open to the world. Examples of services and ports that are inspected include SSH (22), Redis (6379), MySQL (3306), and Microsoft RDP (3339).

You can customize this Bot to look at access list action (allow/deny), direction (ingress/egress), IP protocol (tcp, udp, icmp), list type (security group/NACL), and specific open ports.

  • Permissions Required: Delete

Bot Configuration

Configuration Details

Scope

Access Lists

Filters

  • Access List Rule Ports
  • Access List Rule Open To World
  • Access List Rule IP Protocol
  • Access List Rule Direction
  • Access List Rule Action

Action (1)

Send Slack Message

Jinja2 Template

{{resource.parent_resource_id}} is exposing public access to vulnerable services. It is recommended that these rules be cleaned up, or this ACL/SG will be deleted in 72 hours.

Action (2)

Instance Stop/Start By Tag Value, or
Schedule Deletion in 72 hours

Cloud Users Without Multi-Factor Authentication (MFA) Enabled

This Bot identifies cloud user accounts that are not set up to leverage multi-factor authentication (MFA) when accessing the native cloud portal.

Allowing access to the native portal with standard credentials can expose the organization to attacks and access by malicious users via brute force, social engineering, and other techniques. MFA can reduce risk by requiring a second form of authentication.

  • Permissions Required: Manage

Bot Configuration

Configuration Details

Scope

Cloud User Accounts

Filter(s)

Cloud User Without MFA Enabled

Action (1)

Send Slack Message

Jinja2 Template

{{resource.get_resource_type()}}: {{resource.get_resource_name()}} in account {{resource. get_organization_service_name()}} with account # {{resource.get_organization_service().account_id}} does not have MFA enabled. Please enable Multi-factor Authentication, or this user will be disabled in 12 hours.

Action (2)

Disable Service User: 12 hours

Clouds Without Global API Accounting (CloudTrail)

This Bot inspects Cloud Accounts for API Accounting services (such as AWS CloudTrail) across all regions. When enabled, this Bot ensures that all cloud activity within the native cloud console and via the programmatic API are captured for audit and tracking purposes.

  • Permissions Required: View

Bot Configuration

Configuration Details

Scope

Clouds

Filter(s)

Clouds Without Global API Accounting

Action (1)

Send Slack Message

Jinja2 Template

Please Enable CloudTrail in {{resource. get_organization_service_name()}} with account # {{resource.get_organization_service().account_id}}
Steps to Enable:

  1. Login to AWS Console (with appropriate permissions to View Identity Access Management Account Settings)
  2. Click on Trails on the left navigation pane
  3. Click Get Started Now, if presented
  • Click Add new trail
  • Enter a trail name in the Trail name box
  • Set the Apply trail to all regions option to Yes
  • Specify an S3 bucket name in the S3 bucket box
  • Click Create
  1. If 1 or more trails already exist, select the target trail to enable for global logging
  • Click the edit icon (pencil) next to Apply trail to all regions
  • Click Yes
  • Click Save

Action (2)

N/A

Snapshots Publicly Available

This Bot identifies Snapshots that are accessible to the public (snapshot, database snapshot). Many dramatic data leaks are related to snapshots and/or storage containers with buckets available to the public.

This Bot will discover potential data exposures as they occur and can be customized to send notifications to appropriate individuals or organizations.

  • Permissions Required: Delete

Bot Configuration

Configuration Details

Scope

Instances

Filters

  • Resource is Exposed to Public
  • Snapshot Accessible to Public (works with Snapshot and DB Snapshot)

Action (1)

Send Slack Message

Jinja2 Template

Snapshot {{resource.snapshot_id}} has been identified as public. This violates our security policy. This snapshot resides in {{resource. get_organization_service_name()}}:{{resource.get_organization_service().account_id}} account. Please lock down this snapshot, or this will be deleted in 72 hours.

Action (2)

Schedule Deletion: 72 hours

Storage Containers Exposing Public Permissions

This Bot identifies storage containers exposing data with permissive access lists. You can specify what permissions you’re looking for (full control, read-only, write) and customize this Bot to meet your needs.

To be slightly more aggressive, this Bot can be configured to use the "Scheduled Deletion" action.

  • Permissions Required: Manage, and/or Delete (if using scheduled deletion)

Bot Configuration

Configuration Details

Scope

Instances

Filters

  • Resource is Exposed to Public
  • Storage Container Exposing Specific Permissions: read, write, full control, read acp, write acp
  • Storage Container Public Access

Action (1)

Send Slack Message

Jinja2 Template

An insecure storage container was identified with public permissions to the world.
The details of this resource are:
{{resource.serialize(indent=2)}}
This storage container will have the public permissions removed in 4 hours.

Action (2)

Cleanup Exposed Storage Container: Set time - 4 hours.

Instance Image ID Audit (aka “Golden Image”): Security

Use this Bot to Identify instances running an image that is not known by the system. This is typically the result of the parent image being deleted. Note that the filter does not search public images and only will search Shared/Private images.

  • Permissions Required: Provision, Manage, Delete

Bot Configuration

Configuration Details

Scope

Instances

Filter(s

Instance Image ID (you will need a data collection of the image IDs that you want to use as the gold standard, e.g., your allow list)

Action (1)

Send Slack Message

Jinja2 Template

Starting and stopping: {{event.resource.get_resource_name()}}. Check scheduled events under bot Schedule -- Instance Scheduler to confirm the events are scheduled. This bot will start and/or stop instances based on the tag value of a specified tag key. Note, tag value must be in UTC and follow HHMM format. For example, "1835" would schedule the lifecycle action to run at 6:35PM UTC. This resource has the following tags:
{{resource.serialize_tags(indent=2)}}

Action (2)

Instance Stop/Start By Tag Value

Instance Scheduler (aka “Sleeper Bot”)

This Bot periodically starts and stops instances at a set time each day. It reduces spend on instances by shutting down and (optionally) restarting instances that do not need to be running 24/7.

You can identify these instances by tagging them with key/value pairs to denote what time they should be shut down or started, daily.

  • Permissions needed: Manage

Bot Configuration

Configuration Details

Scope

Instances

Filter(s)

Resource Contains Tag Keys: shutdown_time and/or startup_time:

Schedule stop/start lifecycle actions on instances based on tag value of a specified tag key.

Note: tag value must be in UTC and follow HHMM format. For example, "1835" would schedule the lifecycle action to run at 6:35PM UTC.

Action (1)

Send Slack Message

Jinja2 Template

Starting and stopping: {{event.resource.get_resource_name()}}. Check scheduled events under bot Schedule -- Instance Scheduler to confirm the events are scheduled. This bot will start and/or stop instances based on the tag value of a specified tag key. This resource has the following tags:
{{resource.serialize_tags(indent=2)}}

Action (2)

Instance Stop/Start By Tag Value

Instance Daily Backup

This Bot creates backups of database instances (snapshot) on a daily basis.

You can identify database instances to back up, and using a separate Bot, you can delete backups older than "x" days. This is an important best practice because when a database instance in AWS gets deleted, all automated backups also get deleted. With a manual backup, you’ll lose your data.

  • Permissions needed: Provision, Manage, Delete

Additional Notes: For the Cleanup Bot, you can evaluate Database snapshots, Resource Age Exceeds (e.g., 30 days), and then use scheduled deletion with a Slack message letting users know it’s going to be deleted because a new snapshot is taken by the Bot every 14 days.

Bot Configuration

Configuration Details

Scope

Instances

Filter(s)

Instance Manual Backup Age: 14 days

Action (1)

Send Slack Message

Jinja2 Template

Database instance {{resource.name}} has had no manual backup within 14 days. This database instance resides in {{resource. get_organization_service_name()}}:{{resource.get_organization_service().account_id}} account and has an endpoint of {{resource.endpoint_address}}.
This resource will have a snapshot taken.

Action (2)

Create Database Instance Snapshot: 3 hours

Instances Averaging Low CPU

This Bot identifies compute instances that have been averaging less than user-defined CPU over the user-defined time window.

For example, you can find instances that are using less than 4% CPU over a period of 14 days. The Bot can execute nightly and send notifications about those resources. In addition, the Bot can schedule an instance downsize the next day and, if not stopped, execute the instance downsize to reduce costs.

  • Permissions Required: View

Bot Configuration

Configuration Details

Scope

Instances

Filter(s)

Instance Averaging Low CPU: Match Instances with an Average CPU usage percentage greater or equal to a user defined value, for user defined number of days.

Action (1)

Send Slack Message

Jinja2 Template

Instance has been averaging low CPU for X (user-defined in filter above) days. Here are the details: {{resource.serialize(indent=2)}}
Please downsize this instance for cost savings.

Action (2)

Downsize instance

Resources with TTL (aka “Temp Resources”): Optimization

This Bot works in coordination with a tagging policy that allows resource owners to specify a “time to live” (TTL).

The Bot inspects resource tags, and when it finds a resource with a TTL tag key, it schedules its deletion in the future for the number of hours in its TTL tag value. An effective tagging policy in conjunction with this Bot can improve your organization’s management of cost.

  • Permissions Required: Delete

Bot Configuration

Configuration Details

Scope

Big Data Instances, Big Data Snapshot, Content Delivery Networks, Database Instance, Database Snapshot, Hypervisor, Instance, Reserved Instance, Load Balancer, Cache Instance, Snapshot, Network Interface, Private Image, Private Network, Private Subnet, Public IP, Access List,Access List Rule, Route Table, Cloud User, Snapshot, SSH Key Pair, Storage Container, Volume

Filter(s)

Resource Contains Tag Keys: TTL

Note: TTL value must follow #d#h#m format for days, hours, and minutes. For example, "2d3h0m" would schedule the Bot to run in two days and three hours.

Action (1)

Send Slack Message

Jinja2 Template

{{resource.get_resource_type()}}: {{resource.get_resource_name()}} in account {{resource. get_organization_service_name()}} with account # {{resource.get_organization_service().account_id}} has a Time to Live Tag. This resource will be deleted based upon TTL value associated with TTL key.
Resource Tags: {{resource.serialize_tags(indent=2)}}

Action (2)

Time To Live Deletion:
(Schedules deletion based upon TTL value associated with TTL key. Note, TTL value must follow #d#h#m format for days, hours, and minutes. For example, "2d3h0m" would schedule the Bot to run in two days and three hours.)

Resources Orphaned: Best Practices

Identifies resources that do not appear to be in use. It looks for detached volumes, access lists without instances, detached public IPs, disconnected network interfaces, detached Internet gateways, abandoned load balancers, SSH key pairs not in use, and more. By identifying these resources, you can clean up your environment and reduce costs.

  • Permissions Required: Delete

Bot Configuration

Configuration Details

Scope

Supports: Volume, Access List, Hypervisor, Load Balancer, SSH Key Pair, Route Table, Public IP, Network Interface, Internet Gateway, Service Group

Filter(s)

  • Resource Orphaned: Supports: Volume, Access List, Hypervisor, Load Balancer, SSH Key Pair, Route Table, Public IP, Network Interface, Internet Gateway, Service Group

Action (1)

Send Slack Message

Jinja2 Template

{{resource.get_resource_type()}}: {{resource.get_resource_name()}} in account {{resource. get_organization_service_name()}} with account # {{resource.get_organization_service().account_id}} is orphaned. Please log in to clean up, or this resource will be deleted in 72 hours.

Action (2)

  • Schedule Deletion: 72 hours

Insights That Make Great Bots

Below are just a few of the InsightCloudSec out-of-the-box Insights we offer that work as a great starting point for your Bot creation. To use any of the Insights mentioned below, simply locate the desired Insight with the name and use the actions menu to create your Bot.

  • In the image below, the search for "SSH" has provided several results; Click on the actions menu to the left of the Insight name to launch the Bot creation process for the Insight you want to use.

  • Details are available for Creating Bots and Managing Bots

Create a Bot from an InsightCreate a Bot from an Insight

Create a Bot from an Insight

Available Insights include:

  • Access List Exposing SSH to World (Security Group): Identifies security groups exposing port 22 to the world (0.0.0.0/0).
  • API Accounting Log Exposed: Identifies storage containers, e.g., AWS S3, that expose API accounting logs, e.g., AWS CloudTrail logs.
  • Cloud Account Password Policy Not CIS-compliant: Identifies cloud accounts with password policies not compliant with Center for Internet Security (CIS) guidelines.
  • Cloud Account Without Global API Accounting: Identifies accounts without API accounting, e.g., AWS CloudTrail, enabled across all regions.
  • Cloud Root Account API Access Key Present: Identifies accounts with API access keys present on the root account.
  • Cloud User Inactive: Identifies cloud users who have not logged in for a period of time.
  • Database Instance Publicly Accessible: Identifies database instances that are accessible to the public.
  • Database Instance Without Recent Snapshot: Identifies database instances without a recent manual snapshot.
  • Instance Exposing SSH to World: Identifies compute instances exposing port 22 to the world.
  • Instance Running 24x7: Identifies instances that have been running 24x7 over a given period of time.
  • Load Balancer Without SSL Listener: Identifies load balancers without an SSL listener.
  • Network Without Traffic Logging: Identifies cloud networks, e.g., AWS VPCs, that do not have network logging enabled.
  • Resource Orphaned: Scans a variety of resources, e.g., volumes, load balancers, public IPs, Internet gateways, etc., to identify orphans.
  • Resource Tagging Policy 101: Identifies resources without the Name, Environment and Owner tag keys.
  • Snapshot Publicly Available: Identifies snapshots that are accessible to the public.
  • Storage Container Exposing Access to World: Identifies storage containers, e.g. AWS S3 buckets, exposing data with permissive access lists.
  • Volume With PHI Unencrypted: Identifies volumes with Protected Health Information (PHI) tags that are not encrypted.

Updated about a month ago

Working with Bots (Best Practices & Examples)


Examples of Formatting and Implementation of Bots to Understand InsightCloudSec Automation

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.