DivvyCloud Scalable Deployment

The Scalable Deployment, as the name implies, deploys DivvyCloud in a flexible, highly available environment. It uses three AWS CloudFormation Templates (CFT) to provision the network, database, and compute infrastructure required to manage and adapt to your cloud accounts.

This deployment architecture is capable of scaling to manage many hundreds of clouds. It does so by placing two interface servers behind a load balancer, setting up back-end support on separate, highly available MySQL and Redis servers, and separating workers onto different horizontally-scalable instances.

This architecture allows you to scale your worker instances as your cloud infrastructure grows and is tolerant of system failures.

Of note, if you are interested in using a simpler, all-in-one CFT approach that installs network, data, and compute infrastructure with default options, you can find that CFT here.

Steps for DivvyCloud Deployment

1.) CloudFormation Management Console

Login as an Admin to the AWS console in the account where you wish to deploy DivvyCloud. Access the CloudFormation Management and select a region where:

  • There is an SSH key pair
  • There is room for a VPC (region limit is typically 5 VPCs)
../_images/cft-console.png

2.) Download Templates

Download our CloudFormation templates

curl -O https://s3.amazonaws.com/get.divvycloud.com/cft/Divvy-CFT-Network.json
curl -O https://s3.amazonaws.com/get.divvycloud.com/cft/Divvy-CFT-Data.json
curl -O https://s3.amazonaws.com/get.divvycloud.com/cft/Divvy-CFT-Workers.json

Or view the CloudFormation template source for Network, Data, and Workers.

3.) Use Network Template

Start by uploading the Network template, which deploys a VPC, two Subnets, an Internet Gateway, and the required Security Groups for the Data and Workers resources.

Of note, if you have your own VPC and Subnets that you would like to use, you can skip to Step 8 where you use the Data CFT.

../_images/cft-upload-network.png

4.) Name Stack and Add CIDR Block

We suggest naming the stack Network as the Data and Workers CFTs, by default, look for network resources deployed by a CFT called Network.

Also by default, the DivvyCloud installation is accessible via `http` and `https` to the world. If you would like to restrict access, you can by providing a CIDR block range limited to your IP address. This value will be used for the Elastic Load Balancer that sits in front of the two Web Servers.

../_images/cft-parameters-network.png

5.) Optional Settings

You may add tags to your DivvyCloud resources if you wish, e.g, “Environment”, “Owner”, “Email Address”. N.b., “Name” is defined in the template.

../_images/cft-tags.png

You may enable Termination Protection so that no one can accidentally delete your DivvyCloud installation.

../_images/cft-termination-protection.png

6.) Create Network Infratructure

Create!

../_images/cft-create.png

7.) Monitor Progress

The CloudFormation template provisions network resources and defines relationship among them to deploy DivvyCloud. You can watch the CloudFormation template progress by watching the Events tab. Once the “stack” is complete, you are ready to deploy the next CFT. This step should take about 2 minutes.

../_images/cft-events.png

8.) Use Data Template

Upload the Data template, which deploys an RDS instance and Redis cluster.

../_images/cft-upload-data.png

9.) Name Stack and Provide Deployment Information

We suggest naming the stack Data as the Workers CFT, by default, looks for data resources deployed by a CFT called Data.

There are several parameters you can specify to customize your RDS and Redis deployments. They are:

  • Instance sizes for RDS and Redis (default sizes provided)
  • RDS username and password (password requirements specified on form)
  • Whether the RDS is Mult-Availability Zone or not
../_images/cft-parameters-data-1.png
  • Optional Security Group Overrides
    • You can specify your own security groups instead of those provided by the Network CFT
    • Workers instance security group
    • ELB security group
  • Optional Subnet Overrides
    • You can specify your own subnets instead of those provided by the Network CFT
    • Subnets 1 and 2
../_images/cft-parameters-data-2.png

10.) Optional Settings; Create Data Infrastructure; Monitor Progress

As before, you may add tags to your DivvyCloud resources if you wish, e.g, “Environment”, “Owner”, “Email Address”. N.b., “Name” is defined in the template.

This step is the longest of the three. It could take 20+ minutes.

Of note, because this CFT deploys the persistence layer, its resources persist if the Stack Template is deleted. This behavior reduces the likelihood of unintentionally deleting DivvyCloud data.

12.) Use Workers Template

Upload the Workers template, which deploys the Autoscaling Groups (ASG) and Elastic Load Balancer (ELB).

../_images/cft-upload-workers.png

13.) Name Stack and Provide Deployment Information

There are many parameters you can specify to customize the remainder of your DivvyCloud deployment. They are:

  • Email address for ASG notifications
  • Instance size (default size provided)
  • SSH Key Pair
  • Optional Scaling Overrides
    • You can scale out workers more easily if they are deployed on their own ASG
    • Number of workers (up to 8)
    • Number of docker containers dedicated to harvesting per worker (up to 8)
../_images/cft-parameters-workers-1.png
  • Optional Proxies
    • If desired, you can add proxy settings to your deployment
    • HTTP proxy setting
    • HTTPS proxy setting
  • Optional Security Group Overrides
    • You can specify your own security groups instead of those provided by the Network CFT
    • Workers instance security group
    • ELB security group
  • Optional Subnet Overrides
    • You can specify your own subnets instead of those provided by the Network CFT
    • Subnets 1 and 2
../_images/cft-parameters-workers-2.png
  • Optional Database Login Overrides
    • You can specify your own database login information instead of the values used by the Data CFT
    • RDS username and password
  • Optional Database and Redis Endpoint Overrides
    • You can specify your own database and Redis endpoints instead of the values output by the Data CFT
    • RDS and Redis endpoints
../_images/cft-parameters-workers-3.png

14.) Optional Settings; Create Data Infrastructure; Monitor Progress

As before, you may add tags to your DivvyCloud resources if you wish, e.g, “Environment”, “Owner”, “Email Address”. N.b., “Name” is defined in the template.

This step should take about 2 minutes.

Of note, because this step is so quick, it is easy to tear down your deployed configuration and relaunch in another to allow you to scale up over time. Your data will persist as part of the Data CFT, so there is very limited downtime.

15.) Access DivvyCloud

After the CloudFormation process is complete, you can navigate to your provisioned Elastic Load Balancer to copy the DNS name. Use that address to access DivvyCloud. (If the Admin Account login does not appear, it is probable that your EC2s have not completed their software installation. Wait a few minutes and try again.)

../_images/cft-elb.png

16.) Create Admin Account

Upon first login, you will be asked to provide information that will be used to create a Domain Admin account and generate a license. You will need to provide the following:

  • Full name
  • Email address
  • Company name
  • Username (does not have to be email address)
  • Password (12 character minimum)

Note, the ‘Create Admin’ button will not activate until you have provided the required information and your password matches.

../_images/cft-admin.png

17.) Login

After creating your Domain Admin account, you will be redirected to the default login screen. Login with the information just provided and you’re in!

../_images/cft-done.png

Using DivvyCloud

When you first log in, you are on the ‘Dashboard’. There is a ‘Setup Checklist’ that guides you on useful first steps, specifically:

After you have completed some or all of the checklist, we recommend that you examine your cloud infrastructure from the Resources section and start to explore.

You can try any of the 200+ filters available to inspect your infrastructure for specific items of interest. You can layer filters to further refine your focus.

Or, perhaps easier, look at the Insights section to see popular filters and filter combinations that are organized by category, e.g., security, cost optimization, etc.

You can try different Insights, customize them as you see fit, save them for reuse, and use them to create ‘one-click’ Bot templates that enable automated action to enforce policy, whether security or cost management.

DivvyCloud Components

The default provisioned components are:

  • EC2
    • 2 m5.large
    • Ubuntu 16.04
  • RDS
    • db.m4.large
    • MySQL 5.7.x
  • ElastiCache
    • cache.t2.medium
    • Redis 4.0.10

DivvyCloud Resiliency and Flexibility

Users access DivvyCloud through an Elastic Load Balancer (ELB) that connects to two EC2 instances that each run a web interface process. Given the limited number of concurrent users, two processes are sufficient to ensure DivvyCloud is accessible with high reliability. These webservers are under an Autoscaling Group (ASG) to maintain desired state.

DivvyCloud relies upon MySQL to store, analyze, and present its data. It uses AWS’s Relational Database Service with optional multi-availability zone deployment to ensure high availability. Instance size is user-selected, so it can handle scale.

DivvyCloud relies upon Redis for fast read-write of its job queues and other ephemeral data. It uses AWS’s Elasticache to ensure high availability. Instance size is user-selected, so it can handle the scale of tracking job queues.

DivvyCloud workers can be deployed under a separate ASG to maintain a desired number of worker processes to carry out its functions. These processes are most affected by scale, so increasing the size of the ASG allows for effective management of more and more cloud accounts. In addition, the harvester worker process can be scaled up or down to keep the harvesting job backlog at the desired level.

Finally, when worker processes are deployed under a separate ASG, the scheduler worker process is deployed under its own ASG. As the “master scheduler”, the scheduler process is crucial to DivvyCloud operations and having it run on its own ASG can improve availability under high scale situations.

Architecture

Below is a diagram that shows the overall architecture of the deployed CFT:

../_images/CFT-topology-18.2.png