InsightCloudSec Docs

Welcome to the InsightCloudSec Docs!

InsightCloudSec by Rapid7 (formerly DivvyCloud) is a Cloud-Native Security Platform that provides real-time analysis and automated remediation for continuous security and compliance for your multi-cloud environment.

For questions reach out to us through [email protected].

Take Me to the Docs!    Release Notes

AWS Scalable Test Drive Deployment

Instructions for Deploying a Scalable InsightCloudSec Test Drive in AWS

📘

Value Names (DivvyCloud vs. InsightCloudSec)

Some components use our former product name (DivvyCloud vs. InsightCloudSec). Updates to the naming of these components will be communicated when changes are made, but note that the name difference does not affect functionality within the product.

This page provides details on AWS deployment for InsightCloudSec in a flexible, highly available environment. It uses three AWS CloudFormation Templates (CFT) to provision the network, database, and compute infrastructure required to manage and adapt to your cloud accounts.

This deployment architecture is capable of scaling to manage many hundreds of clouds. It does so by placing two interface servers behind a load balancer, setting up back-end support on separate, highly available MySQL and Redis servers, and separating workers onto different horizontally-scalable instances.

This architecture allows you to scale your worker instances as your cloud infrastructure grows and is tolerant of system failures.

Steps for InsightCloudSec Deployment

1. CloudFormation Management Console

Login as an Admin to the AWS console in the account where you wish to deploy InsightCloudSec. Access the CloudFormation Management and select a region where:

  • There is an SSH key pair
  • There is room for a VPC (region limit is typically 5 VPCs)

2. Download Templates

Download our CloudFormation templates or deploy them directly from here:

JSON Format:

curl -O https://s3.amazonaws.com/get.divvycloud.com/cft/Divvy-CFT-Network.json
curl -O https://s3.amazonaws.com/get.divvycloud.com/cft/Divvy-CFT-Data.json
curl -O https://s3.amazonaws.com/get.divvycloud.com/cft/Divvy-CFT-Workers.json

YAML Format:

curl -O https://s3.amazonaws.com/get.divvycloud.com/cft/Divvy-CFT-Network.yaml
curl -O https://s3.amazonaws.com/get.divvycloud.com/cft/Divvy-CFT-Data.yaml
curl -O https://s3.amazonaws.com/get.divvycloud.com/cft/Divvy-CFT-Workers.yaml

You can view the CloudFormation JSON template source for Network, Data, and Workers.

Or view the CloudFormation YAML template template source for Network, Data, and Workers.

Open in a new tab to keep this page open

Network Stack:

Data Stack:

Workers Stack:

3. Use Network Template

Start by uploading the Network template, which deploys a VPC, two Subnets, an Internet Gateway, and the required Security Groups for the Data and Workers resources.

Of note, if you have your own VPC and Subnets that you would like to use, you can skip to Step 8 where you use the Data CFT.

4. Name Stack and Add CIDR Block

We suggest naming the stack Network as the Data and Workers CFTs, by default, look for network resources deployed by a CFT called Network.

Also by default, the InsightCloudSec installation is accessible via http and https to the world. If you would like to restrict access, you can by providing a CIDR block range limited to your IP address. This value will be used for the Elastic Load Balancer that sits in front of the two Web Servers.

5. Optional Settings

You may add tags to your InsightCloudSec resources if you wish, e.g, “Environment”, “Owner”, “Email Address”. N.b., “Name” is defined in the template.

Note: You may want to enable Termination Protection so that no one can accidentally delete your InsightCloudSec installation.

6. Create Network Infrastructure

Create!

7. Monitor Progress

The CloudFormation template provisions network resources and defines relationship among them to deploy InsightCloudSec. You can watch the CloudFormation template progress by watching the Events tab. Once the “stack” is complete, you are ready to deploy the next CFT. This step should take about 2 minutes.

8. Create Data Stack

Upload the Data template, which deploys an RDS instance and Redis cluster. Click the "Create Stack" button on the CloudFormation main page. Select the source to either point to the Data S3 URL listed above, or choose the locally downloaded version instead.

9. Name Stack and Provide Deployment Information

We suggest naming the stack Data as the Workers CFT, by default, looks for data resources deployed by a CFT called Data.

There are several parameters you can specify to customize your RDS and Redis deployments. They are:

  • Instance sizes for RDS and Redis (default sizes provided)
  • RDS username and password (password requirements specified on form)
  • Whether the RDS is Mult-Availability Zone or not

Optional Security Group Overrides

  • You can specify your own security groups instead of those provided by the Network CFT
  • Workers instance security group
  • ELB security group

Optional Subnet Overrides

  • You can specify your own subnets instead of those provided by the Network CFT
  • Subnets 1 and 2

10. Optional Settings; Create Data Infrastructure; Monitor Progress

As before, you may add tags to your InsightCloudSec resources if you wish, e.g, “Environment”, “Owner”, “Email Address”. N.b., “Name” is defined in the template.

This step is the longest of the three. It could take 20+ minutes.

Of note, because this CFT deploys the persistence layer, its resources persist if the Stack Template is deleted. This behavior reduces the likelihood of unintentionally deleting InsightCloudSec data.

11. Create Workers Stack

Upload the Workers template, which deploys the Autoscaling Groups (ASG) and Elastic Load Balancer (ELB). Click the "Create Stack" button on the CloudFormation main page. Select the source to either point to the Workers S3 URL listed above, or choose the locally downloaded version instead.

12. Name Stack and Provide Deployment Information

There are many parameters you can specify to customize the remainder of your InsightCloudSec deployment. They are:

  • Email address for ASG notifications
  • Instance size (default size provided)
  • SSH Key Pair
  • Optional Scaling Overrides
  • You can scale out workers more easily if they are deployed on their own ASG
  • Number of workers (up to 8)
  • Number of docker containers dedicated to harvesting per worker (up to 8)

Optional Proxies

  • If desired, you can add proxy settings to your deployment
  • HTTP proxy setting
  • HTTPS proxy setting

Optional Security Group Overrides

  • You can specify your own security groups instead of those provided by the Network CFT
  • Workers instance security group
  • ELB security group

Optional Subnet Overrides

  • You can specify your own subnets instead of those provided by the Network CFT
  • Subnets 1 and 2

Optional Database Login Overrides

  • You can specify your own database login information instead of the values used by the Data CFT
  • RDS username and password

Optional Database and Redis Endpoint Overrides

  • You can specify your own database and Redis endpoints instead of the values output by the Data CFT
  • RDS and Redis endpoints

13. Optional Settings; Create Data Infrastructure; Monitor Progress

As before, you may add tags to your InsightCloudSec resources if you wish, e.g, “Environment”, “Owner”, “Email Address”. N.b., “Name” is defined in the template.

This step should take about 2 minutes.

Of note, because this step is so quick, it is easy to tear down your deployed configuration and relaunch in another to allow you to scale up over time. Your data will persist as part of the Data CFT, so there is very limited downtime.

14. Access InsightCloudSec

After the CloudFormation process is complete, you can navigate to your provisioned Elastic Load Balancer to copy the DNS name. Use that address to access InsightCloudSec. (If the Admin Account login does not appear, it is probable that your EC2s have not completed their software installation. Wait a few minutes and try again.)

15. Create Admin Account

Upon first login, you will be asked to provide information that will be used to create a Domain Admin account and generate a license. You will need to provide the following:

  • Full name
  • Email address
  • Username (does not have to be email address)
  • Password (12 character minimum)
  • Company name
  • Base URL

Note, the "Create Admin and Login" button will not activate until you have provided the required information and your password matches.

16. Login

After creating your Domain Admin account, you will be redirected to the default login screen. Login with the information just provided and you’re in!

InsightCloudSec Components

The default provisioned components are:

EC2

  • 2 m5.large
  • Ubuntu 18.04

RDS

  • db.m4.large
  • MySQL 5.7.x

ElastiCache

  • cache.t2.medium
  • Redis 4.0.10

InsightCloudSec Resiliency and Flexibility

Users access InsightCloudSec through an Elastic Load Balancer (ELB) that connects to two EC2 instances that each run a web interface process. Given the limited number of concurrent users, two processes are sufficient to ensure InsightCloudSec is accessible with high reliability. These webservers are under an Autoscaling Group (ASG) to maintain desired state.

InsightCloudSec relies upon MySQL to store, analyze, and present its data. It uses AWS’s Relational Database Service with optional multi-availability zone deployment to ensure high availability. Instance size is user-selected, so it can handle scale.

InsightCloudSec relies upon Redis for fast read-write of its job queues and other ephemeral data. It uses AWS’s Elasticache to ensure high availability. Instance size is user-selected, so it can handle the scale of tracking job queues.

InsightCloudSec workers can be deployed under a separate ASG to maintain a desired number of worker processes to carry out its functions. These processes are most affected by scale, so increasing the size of the ASG allows for effective management of more and more cloud accounts. In addition, the harvester worker process can be scaled up or down to keep the harvesting job backlog at the desired level.

Finally, when worker processes are deployed under a separate ASG, the scheduler worker process is deployed under its own ASG. As the “master scheduler”, the scheduler process is crucial to InsightCloudSec operations and having it run on its own ASG can improve availability under high scale situations.

Architecture

Below is a diagram that shows the overall architecture of the deployed CFT:

Updated 15 days ago

AWS Scalable Test Drive Deployment


Instructions for Deploying a Scalable InsightCloudSec Test Drive in AWS

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.