Compliance Exporting

We recommend using the Compliance Scorecard instead

While the process documented for Compliance Exporting is still supported, we strongly recommend users take advantage of the capabilities and functionality that are included with the Compliance Scorecard for better data visibility, reporting, and context around compliance.

InsightCloudSec offers the export of compliance information from Insight packs. The instructions below include steps to set up a report that is generated daily in the early hours of the morning UTC time and uploaded to the user's AWS or GCP storage bucket of choice using the credentials of an organization service account.

The contents of this report provide low level data mapping of resources to compliance issues. The content is intended for teams wanting to use the data for custom reports and integrations with external business intelligence and analytics tools. Linking one or more packs to an export configuration selects the organization service account for credentials and provides a bucket name and filename prefix for objects.

Bucket and Permissions

The bucket name provided must exist and the selected account must have object write permissions for that bucket. Buckets and permissions are not validated when an export configuration is created.

Create an Export Configuration using a storage container

  1. Select a pack to link to an export configuration by clicking the link icon on the pack card.
  2. Click Add Link. If there are no export configurations or if all of your links are already associated with configurations, you will be prompted to create a new configuration.
  3. Click "Create Configuration" to access the "Manage Configurations" tab. This form can be used for creating or editing configurations by selecting an existing configuration or by completing the form to create a new one.

Once created, the configuration will be automatically linked to the selected pack.

Run your report

Once your configuration is created, you have the option to run the export report immediately. This is rate limited at one request per five minutes. We recommend running the report at least once to verify the bucket configuration is correct. Depending on the size of the installation and number of non-compliant resources, the report can take several minutes to run and upload to the bucket.

To run the report immediately, click the blue Run button.

From the Manage Export Configurations Links modal, you can also edit or unlink the configuration from the pack. To edit the configuration, select the blue button with the pencil. To unlink the configuration, select the middle button with the link.

Once a configuration is complete, you may link other packs to it. From the Insight Packs page, select the link icon on another pack you would like to link to an existing configuration. The following illustrates the modal that appears. The list of your existing configurations appears in the drop-down list box. Select one of these and select 'Add'.

Report contents

The report consists of two types of files. The first is a single report.json file which acts as a header file that supplies all of the meta data about packs, insights, resource types, and cloud accounts. The remaining files are the json-serialized non-compliant resources by insight.

1
.
2
└── myprefix-data-export-2019-02-21T18:40:18Z
3
├── backoffice-11.json
4
├── ... clip
5
├── backoffice-90.json
6
├── custom-91.json
7
├── custom-98.json
8
└── report.json

Storage Buckets

Blob storage services like Amazon's S3 and Google's Storage Buckets do not have directories/folders for organizing objects. By convention, object names can have prefixes that look like file system paths. For example, /reports/year/month/day/report.json. This report names all objects with the standard format of <prefix>-data-export-<timestamp>/<file_name>.json, where prefix is supplied by the user.

Insight files are named with the following convention; <insight_name>:<insight_source (backoffice | custom)>:<insight_id>.

Header file: report.json

The report.json file provides all the metadata necessary to programmatically understand the output in the insight files. The file is formatted as a single json object with six keys:

  • name: Name of the report
  • version
  • resource_types: object where keys are resource_type name and value is an object of property names used in serialization and their type.
  • organization_services: list of the name, organization_service_id, and account_id for all organization services.
  • packs: list of packs. Each pack contains compliance metadata from their respective framework (e.g., HIPPA). It also lists the insights by ID that make up the pack. Insight IDs are grouped by source in two lists under the keys backoffice and custom. The source and ID are needed to look up insights.
  • insights: object where key is <source>:<insight_id> and value is a serialized insight. The insight also has a count key that breaks down results and totals by organization_service_id.

Insight Sources

InsightCloudSec has a repository of Insights we serve from an internal service that customer installs harvest down. These are referenced by their source, backoffice, and their ID, which is an integer.

Insights created in the customer install have the source of custom and an ID which is an integer.

Insight IDs are unique to their source. Use caution to ensure you are referencing the correct source. In the report.json header file packs list their insights IDs by source.

Insight resources

A file is created for each insight and resources are serialized and written per line. This allows for lazy line-by-line reading for large datasets. Each line consists of a json object with two keys. The key resource_type is the InsightCloudSec internal resource type identifier and can be used to look up the properties of the serialization in the report.json file at the path report["resource_types"]["bigdatainstance"]. The resource key contains the resource serialization following the format in the header file.

1
{
2
"resource": {
3
"instance_flavor_resource_id": "instanceflavor:123:ap-southeast-1:bigdata:ds2.xlarge:",
4
"db_name": null,
5
"organization_service_id": 123,
6
"name": "bi",
7
"endpoint_address": "bi.cwqmjcawzudl.ap-southeast-1.redshift.amazonaws.com",
8
"resource_id": "bigdatainstance:123:ap-southeast-1:bi:",
9
"encrypted": false,
10
"availability_zone": "ap-southeast-1a",
11
"ssl_required": false,
12
"instance_id": "bi",
13
"instance_type": "ds2.xlarge",
14
"state": "available",
15
"version": "1.0",
16
"publicly_accessible": false,
17
"master_username": "root",
18
"backup_retention": 1,
19
"endpoint_port": 54xx,
20
"key_resource_id": null,
21
"id": 1,
22
"tags": {
23
"customer": "sample-international",
24
"application": "sample-apac",
25
"contact-email": "first.last@sample.com",
26
"environment": "prod",
27
"team": "sampleteam-international"
28
},
29
"region_name": "ap-southeast-1"
30
},
31
"resource_type": "bigdatainstance"
32
}

Tags

Note that the tags key is not provided in the resource type metadata. That is because tags are stored in a generic way and can always be assumed to be of the type {string: string}.

Export via email notification

Manage subscriptions (Notifications)

Email notifications (or subscriptions) associated with Insight packs are available for administration through the Insights landing page.

To read more about setting up SMTP, configuring email notifications, and creating pack-level notifications, check out our documentation on SMTP (Email Notifications).

After selecting the specific Insight Pack, you can select the envelope icon (Manage Subscriptions). This icon opens a page that provides a list of all subscriptions associated with the Insight Pack.

From this page you can add a new subscription, or modify an existing subscription by clicking on the ellipsis menu to display the following options:

  • Send Now - Send a new report based on your Insight Pack immediately (On-Demand Report)
  • Edit - modify or update your subscription
  • Delete - remove your subscription