Job Backlog Export

In the "System Administration" section of InsightCloudSec, customers can view important System Health Information like the Current Job Backlog. The Current Job Backlog is a great way to understand the current load on your environment.

Exporting these Job Backlog metrics can be used to drive improvements from a scaling perspective and provide historical data for up to 15 months. InsightCloudSec provides support for export of Job Backlog metrics to AWS through CloudWatch and Google Cloud Platform through Stackdriver.

📘
Items to Note

Content on this page may reference either DivvyCloud (in examples or provided content) or InsightCloudSec, but the functionality is the same.

If you have questions or need assistance with the capabilities described here reach out to us through the Customer Support Portal.

AWS CloudWatch Job Backlog Export

The following instructions are for setting up export for AWS and CloudWatch.

❗️
Required Permissions
Update the IAM policy for your account to allow for "cloudwatch:PutMetricData".

Setting Up Export

1. From the Administration Section of the left-side panel navigation, select "System Administration --> System" and scroll to the Job Backlog Settings.

2. Under Target Account, Admins can select the AWS Account that will receive exported job backlog information.

You will need the name of the target account and target region for export.
You also have the option to use instance authentication if you're using Instance Assume Role. For most cases, you will not need to use instance authentication.
To include a custom name, fill in the "Target Namespace (AWS Only)" field, otherwise this will display as InsightCloudSec (or DivvyCloud) in CloudWatch.

Viewing Details in the AWS Console

In the AWS console, this information will show up in your 'Custom Namespaces', under Metrics. You will have many options for how this data will be displayed.

You have the choice of viewing time series data from 1 hour up to 15 months.
In the example below, entries are being pushed every 5 minutes but that can be modified.

GCP Stackdriver Job Backlog Export

The following instructions are for setting up export for GCP and Stackdriver.

❗️
Required Permissions
To enable access to the logging data in a GCP project, you must be a project member and have a Cloud IAM role within GCP that grants permission for logging. The required permissions are:

“monitoring.metricDescriptors.create”

“monitoring.timeSeries.create”

Permissions can be given to a new role or an existing role under the "IAM & Admin --> IAM" section of the GCP Dashboard.

Prerequisites (GCP)

In order for InsightCloudSec to provide the appropriate details, you will need:

A connected GCP cloud
A GCP role with appropriate permissions

To enable the Stackdriver Monitoring API, complete the following steps to enable or confirm.

1. In the GCP Console, access the main left-panel navigation menu. Under API & Services select "Library".

2. Search for "Stackdriver" and choose the option for "Stackdriver API".

3. From the Stackdriver API summary page, select "Enable".

If these settings are already enabled, you will see a green tick and a "Manage" option and no further action is required.

Setting Up Export (in InsightCloudSec)

1. From the Administration Section of the left-side panel navigation, select "System Administration --> System" and scroll to the Job Backlog Settings.

2. Under Target Account, select the GCP Account from the drop-down that will receive exported job backlog information; save your selection.

Viewing Details in the GCP Console

1. From the main GCP console, search for the "Stackdriver" service, select "Stackdriver" from the dropdown result, and click to "Go to Stackdriver".

2. From the left panel, click on Dashboards, click "Create Dashboard", and select "Add Chart".

3. Give your chart a useful title. Then in "Find resource type and metric", search for "divvy" to find these three metrics:

job_backlog_standard
job_backlog_high_priority
job_backlog_immediate

4. Select your desired metric, click on "Add Metric". Repeat this for each one and then select "Save".

5. Verify that your data displays appropriately.

Note: This can take up to a minute or more.