DivvyCloud

Welcome to the DivvyCloud Docs!

DivvyCloud is a Cloud Security Posture Management (CSPM) platform that provides real-time analysis and automated remediation across leading cloud and container technologies.

For questions about documentation reach out to us [email protected]

Take Me to the Docs!    Release Notes

Event-Driven Harvesting (AWS)

Overview

DivvyCloud has the capability to augment standard, polling-based harvestingharvesting - is the term used to describe how DivvyCloud collects data from the cloud service providers. with Event-Driven Harvesting (EDH). EDH pulls data from AWS CloudWatch Events and AWS CloudTrail into a central event bus for consumption by the platform.

This modern approach to data collection not only improves DivvyCloud's cadence for providing resource visibility and opportunities for remediation, but also enriches the data with lifecycle changes enabling auditing capabilities. With this EDH-provided data, identifying how a resource entered a noncompliant state becomes much easier at scale.

Basic EDH Structure

Prerequisites

Before you get started with EDH you will need to ensure you have the following:

  • A functioning DivvyCloud platform with the appropriate Admin (Org or Domain) permissions.

As usual if you have questions or encounter issues, reach out us at [email protected].

Concepts

EDH relies on a relationship between a Producer and Consumer.

  • A Producer is an AWS account that is configured to forward cloud events to another account.
  • A Consumer is a central account that ingests events from one or more Producers. Consumers also create their own events and are essentially Producers as well.

EDH uses an Amazon SQS topic to retrieve the consumed events on a region-by-region basis.
Customers are encouraged to set up at least two Consumers, one to consume data from development environments and another to consume data from the production environment. These two types of accounts require specific IAM permissions (see Requirements).

Steps to Configure

Configure Consumer Account

Consumer accounts are tasked with ingesting events from other accounts and also producing their own events (i.e., consumers are both producers and consumers).

📘

Opt-In Regions

If you are interested in disabling opt-in regions from harvesting, you can include that configuration when you establish the settings within your consumer(s). If you need assistance with this contact us through [email protected].

Refer to additional documentation on Opt-In Regions here.

1. Login to your AWS Console and open the IAM dashboard.

2. Open the "Policies" section and create a new policy using the "JSON" tab to assign to the existing user/role that DivvyCloud uses to access the consumer account.

  • This policy allows the DivvyCloud user to read and write the appropriate event data.
  • In this case, our user is 'DivvyCloud-PowerUser' and our policy name is 'DivvyCloud-EDH-Consumer'.

In addition, you can review the AWS Standard & Power User Policies for details.

Here is a sample of what the policy looks like:

{
    "Version": "2012-10-17",
    "Statement": [
{
"Action": [
                "events:DeleteRule",
                "events:DescribeEventBus",
                "events:DescribeRule",
                "events:PutPermission",
                "events:PutRule",
                "events:PutTargets",
                "events:RemoveTargets",
                "iam:GetPolicy",
                "iam:GetPolicyVersion",
                "iam:GetRole",
                "iam:ListPolicies",
                "iam:ListPolicyVersions",
                "organizations:DescribeOrganization",
                "sqs:ListQueues",
                "sqs:CreateQueue",
                "sqs:DeleteQueue",
                "sqs:ReceiveMessage",
                "sqs:SetQueueAttributes",
                "sqs:DeleteMessage",
                "sqs:DeleteMessageBatch"
            ],
            "Effect": "Allow",
            "Resource": "*"
} ]
}

And here is the Create Policy interface in the AWS Console

Create Policy in AWS

3. Click on the "Review Policy" option to add the Name, Description, and to review the Summary.

  • In this example we've named our policy 'DivvyCloud-EDH-Consumer' to identify it as the 'consumer' policy.

Review Consumer Policy

4. Locate your DivvyCloud Power User. Open the permissions tab to attach the newly created consumer policy to your DivvyCloud Power User (e.g., 'DivvyCloud-PowerUser' below).

Attach the EDH Consumer Policy

❗️

If you already have at least one CloudTrail enabled with at least Write events being logged, you don't need to create an additional trail for this and can skip to Step 7.

5. With our consumer policies in place, you will need to configure CloudTrail. In the AWS console, open "Services" --> "CloudTrail"

  • In this example, the trail is named 'DivvyCloud-EDH'.

CloudTrail Dashboard

6. Click on "Trails" --> "Create Trail".

Create a New Cloud Trail

7. Navigate to "CloudWatch" --> "Events" --> "Event Buses". Click on the "Add permission" button and enable Organization access to default event bus.

Event Bus Permissions

8. If organization permissions do not already exist, add them as follows:

Add Permission to Event Bus

Add Permission to Event Bus

Configure Producer Account

1. Login to your AWS Console and open the IAM dashboard.

2. Open the "Policies" section and create a new policy using the "JSON" tab. This new policy will be assigned to the existing user/role that DivvyCloud uses to access the producer account.

  • This policy allows the DivvyCloud user to write the appropriate event data.
  • In this case, our user is 'DivvyCloud-PowerUser' and our policy name is 'DivvyCloud-EDH-Producer'.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "events:DeleteRule",
                "events:DescribeRule",
                "events:PutRule",
                "events:PutTargets",
                "events:RemoveTargets",
                "iam:CreatePolicyVersion",
                "iam:DeletePolicyVersion",
                "iam:GetPolicy",
                "iam:GetPolicyVersion",
                "iam:GetRole",
                "iam:PassRole",
                "organizations:DescribeOrganization"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

3. Create a second policy which will allow the automated creation of roles and permissions and allow the producers to feed our consumer.

  • This policy will be named 'DivvyCloud-EDH-AutoRole’.
  • Replace <account_id> with the ID of the current AWS account.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:AttachRolePolicy",
                "iam:PassRole"
            ],
            "Resource": "arn:aws:iam::<account_id>:role/service-role/event-driven-harvest/*eventbus-role"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreatePolicy",
                "iam:CreatePolicyVersion",
                "iam:DeletePolicyVersion"
            ],
            "Resource": "arn:aws:iam::<account_id>:policy/service-role/event-driven-harvest/*eventbus-policy"
        }

    ]
}

Manually Configure Producer Account (Alternative)

1. Alternatively, you can manually create a role and policy which grants the consumer access to the producer’s logs. This role/policy will need be to created on every producer account.

2. Save the following JSON for use with the the ‘assume-role-policy-document’. Save as ‘edh_role_manual_sts.txt’.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {"Service": "events.amazonaws.com"},
            "Action": "sts:AssumeRole"
        }
    ]
}

3. Create divvycloud-eventbus-role.

aws iam create-role --role-name divvycloud-eventbus-role --assume-role-policy-document file:///PATH/TO/FILE/edh_role_manual_sts.txt --path /service-role/event-driven-harvest/
{
    "Role": {
        "Path": "/service-role/event-driven-harvest/",
        "RoleName": "divvycloud-eventbus-role",
        "RoleId": "AROAIYD6PGH3VPCBX4D4Q",
        "Arn": "arn:aws:iam::ACCT-ID-HERE:role/service-role/event-driven-harvest/divvycloud-eventbus-role",
        "CreateDate": "2018-11-02T20:27:37Z",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "events.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}

4. Save the following JSON for use as the ‘policy-document’.

  • Be sure to update the ‘CONSUMER-ACCT-ID’ value with the account ID of the consumer account.
  • Save as ‘edh_role_manual_policy.txt’
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "events:PutEvents"
            ],
            "Resource": [
                "arn:aws:events:*:CONSUMER-ACCT-ID:event-bus/default"
            ],
            "Effect": "Allow"
        }
    ]
}

5. Create divvycloud-eventbus-policy.

aws iam create-policy --policy-name divvycloud-eventbus-policy --policy-document file:///PATH/TO/FILE/edh_role_manual_policy.txt
{
    "Policy": {
        "PolicyName": "divvycloud-eventbus-policy",
        "PolicyId": "ANPAJ52WSKTB7L4W3QXFS",
        "Arn": "arn:aws:iam::CONSUMER-ACCT-ID:policy/divvycloud-eventbus-policy",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 0,
        "IsAttachable": true,
        "CreateDate": "2018-11-02T20:42:27Z",
        "UpdateDate": "2018-11-02T20:42:27Z"
    }
}

6. Attach divvycloud-eventbus-policy to divvycloud-eventbus-role.

aws iam attach-role-policy --role-name divvycloud-eventbus-role --policy-arn arn:aws:iam::ACCT-ID:policy/divvycloud-eventbus-policy

aws iam list-attached-role-policies --role-name divvycloud-eventbus-role
{
    "AttachedPolicies": [
        {
            "PolicyName": "divvycloud-eventbus-policy",
            "PolicyArn": "arn:aws:iam::ACCT-ID:policy/divvycloud-eventbus-policy"
        }
    ]
}

7. DivvCloud will automatically use the divvycloud-eventbus-role; the role does not need to be associated with ‘DivvyCloud-PowerUser'.

Complete the Setup of Your Producer

1. Now that you have created your producer policies you will need to attach them to your DivvyCloud-PowerUser in one of two ways:

Attach Policies

2. Or, for users that manually configured the producer, as follows:

Attach Policy (Manual Alternate)

❗️

If you already have at least one CloudTrail enabled with at least Write events being logged, you don't need to create an additional trail for this and can skip to Step 7.

3. With your producer policies in place, you can now configure CloudTrail. Navigate to "Services" --> "CloudTrail".

  • In this example, the trail is named ‘DivvyCloud-EDH’.

Configure CloudTrail

4. Click on "Trails" --> "Create Trail".

Create CloudTrail

Configure EDH in DivvyCloud

After completing the AWS configuration, you are now ready to complete the configuration of EDH inside of DivvyCloud.

1. Navigate to the Clouds main page (under Cloud on the left-side navigation menu).

Clouds Page

2. If no EDH has been set up previously, select Begin and then confirm the enabling of EDH. Otherwise, go to Step 3.

New EDH Setup

Enable EDH Dialog

3. From the Event-Driven Harvesting tab, select "Consumer" and then select "Add Consumer".

EDH - Add Consumer

4. Select your Consumer from the drop-down list.

Add Event Consumer

  • The Consumer will remain in a pending state while setup is in progress.

Consumer "pending"

❗️

Enabled Note

Do not add producers until the consumer is showing as enabled.

Consumer "enabled"

5. From the Event-Driven Harvesting tab, select the Producers sub tab and click "Add Producer".

EDH - Add Producer

6. Complete the "Enable Producer" form with the details from the AWS setup completed earlier.

EDH - Add Producer Form

Note: If you have questions about the policy associated with enabling any options on this form contact [email protected]

  • You can choose to automatically enable event support for all future resource types as they become supported by DivvyCloud (This is recommended).
  • You can also set EDH to "Automatically Provision IAM resources" and "Automatically Provision Cloudwatch resources"

7. After adding your producer, it may remain in the pending state for several minutes while setup is in progress.

EDH - Add Producer "pending"

8. Once the configuration is complete, the producer account’s status will be labeled enabled.

📘

Enabling Producer

If there are any issues enabling your producer, the status will read "error" and you’ll be able to click the read icon to see related errors.

9. If you added a heavily used account as your producer, you should see events within a minute or so.

  • If not, you can create and delete a test user to generate events. In this case, we’ll create a user named ‘EDH-TEST-DELETE-ME’ to generate events and verify your configuration.

EDH - Test User

Considerations

Below are some additional details that you may want to review as part of understanding your EDH setup.

DivvyCloud sets a maximum harvesting frequency of four hours for AWS EDH-enabled resources and regions.

EDH Supported Resources

AutoScaling Groups

AutoScaling Launch Config

Broker MQ Instances

CFT

EBS Snapshots

EBS Volumes

EC2/VPC Instances

ElasticSearch

Encryption Keys (KMS)

HyperVisor

IAM Groups

IAM Password Policy

IAM Policies

IAM Roles

IAM Users

Identity Provider

Internet Gateways

Load Balancer

Memcache/Redis

NAT Gateways

Network Interface

Placement Group

Private Image

RDS Cluster

RDS Snapshot

RDS

RedShift

Route Tables

S3

SNS Subscription

SNS Topic

SSH Keypairs

Security Groups and Network ACLs

Service Access Key

Subnets

VPC Flow Logs

VPC Network Peers

VPCs

The following events are currently configured to publish to Consumers.

Resource Type:
    SupportedEvent
    
AutoScaling Groups:
    AttachInstances
    CreateAutoScalingGroup
    CreateOrUpdateTags
    DeleteAutoScalingGroup
    DetachInstances
    PutScalingPolicy
    SetDesiredCapacity
    SetInstanceProtection
    UpdateAutoScalingGroup
    
AutoScaling Launch Config:
    CreateLaunchConfiguration
    DeleteLaunchConfiguration
    
Broker MQ Instances:
   CreateBroker
   DeleteBroker
   UpdateBroker

CFT:
    CancelUpdateStack
    CreateStack
    DeleteStack
    UpdateStack
    UpdateTerminationProtection
    
EBS Snapshots:
    CreateSnapshot
    CreateTags
    DeleteSnapshot
    DeleteTags
    ModifySnapshotAttribute
    
EBS Volumes:
    AttachVolume
    CreateTags
    CreateVolume
    DeleteTags
    DeleteVolume
    DetachVolume
    ModifyVolume
    ModifyVolumeAttribute    
    
EC2/VPC Instances:
    AssociateIamInstanceProfile
    CreateTags
    DeleteTags
    DisassociateIamInstanceProfile
    ModifyInstanceAttribute
    MonitorInstances
    RebootInstances
    RunInstances
    StartInstances
    StopInstances
    TerminateInstances
    UnmonitorInstances
    
ElasticSearch:
    CreateElasticsearchDomain
    DeleteElasticsearchDomain
    UpgradeElasticsearchDomain
    UpdateElasticsearchDomainConfig
    
Encryption Keys (KMS):
    CreateKey
    DisableKey
    DisableKeyRotation
    EnableKey
    EnableKeyRotation
    PutKeyPolicy
    TagResource
    UntagResource
    UpdateKeyDescription
    
HyperVisor:
    AllocateHosts
    ModifyHosts
    ReleaseHosts
    
IAM Groups:
    AttachGroupPolicy
    CreateGroup
    DeleteGroup
    DeleteGroupPolicy
    DetachGroupPolicy
    PutGroupPolicy
    
IAM Password Policy:
    DeleteAccountPasswordPolicy
    DeleteAccountPublicAccessBlock
    PutAccountPublicAccessBlock
    UpdateAccountPasswordPolicy    

IAM Policies:
    CreatePolicy
    CreatePolicyVersion
    CreateSAMLProvider
    DeleteAccountPasswordPolicy
    DeleteAccountPublicAccessBlock
    DeletePolicy
    DeletePolicyVersion
    DeleteSAMLProvider
    PutAccountPublicAccessBlock
    UpdateAccountPasswordPolicy

IAM Roles:
    AttachRolePolicy
    CreateRole
    DeleteRole
    DeleteRolePermissionsBoundary
    DeleteRolePolicy
    DetachRolePolicy
    PutRolePermissionsBoundary
    PutRolePolicy
    TagRole
    UntagRole
    UpdateAssumeRolePolicy
    
IAM Users:
    AddUserToGroup
    AttachUserPolicy
    CreateLoginProfile
    CreateUser
    CreateVirtualMFADevice
    DeactivateMFADevice
    DeleteLoginProfile
    DeleteUser
    DeleteUserPolicy
    DeleteVirtualMFADevice
    DetachUserPolicy
    EnableMFADevice
    PutUserPolicy
    RemoveUserFromGroup
    TagUser
    UntagUser
    UpdateUser
    
Identity Provider:
    CreateSAMLProvider
    DeleteSAMLProvider
    
Internet Gateways:
    AttachInternetGateway
    CreateInternetGateway
    DeleteInternetGateway
    DetachInternetGateway
    
Load Balancer:
    AddTags
    ApplySecurityGroupsToLoadBalancer
    AttachLoadBalancerToSubnets
    CreateLoadBalancer
    CreateLoadBalancerListeners
    DeleteLoadBalancer
    DeleteLoadBalancerListeners
    DeregisterInstancesFromLoadBalancer
    DetachLoadBalancerFromSubnets
    ModifyLoadBalancerAttributes
    RegisterInstancesWithLoadBalancer
    RemoveTags
    SetSecurityGroups
    SetSubnets
    
Memcached/Redis:
    CreateCacheCluster
    CreateReplicationGroup
    DeleteCacheCluster
    DeleteReplicationGroup
    ModifyCacheCluster
    ModifyReplicationGroup
    RebootCacheCluster
    
NAT Gateways:
    CreateNatGateway
    DeleteNatGateway
    
Network Interface:
    CreateNetworkInterface
    DeleteNetworkInterface
    ModifyNetworkInterfaceAttribute
    
Placement Group:
    CreatePlacementGroup
    DeletePlacementGroup
    
Private Image:
    CreateImage
    ImportImage
    RegisterImage
    DeregisterImage
    
RDS Cluster:
    CreateDBCluster
    DeleteDBCluster
    ModifyDBCluster
    StartDBCluster
    StopDBCluster
    
RDS Snapshot:
    AddTagsToResource
    CreateDBClusterSnapshot
    CreateDBSnapshot
    DeleteDBClusterSnapshot
    DeleteDBSnapshot
    RemoveTagsFromResource
    
RDS:
    AddTagsToResource
    CreateDBInstance
    CreateDBInstanceReadReplica
    DeleteDBInstance
    ModifyDBInstance
    RebootDBInstance
    RemoveTagsFromResource
    StartDBInstance
    StopDBInstance
    
RedShift:
    AuthorizeSnapshotAccess
    BatchDeleteClusterSnapshots
    CreateCluster
    CreateClusterSnapshot
    CreateTags
    DeleteCluster
    DeleteClusterSnapshot
    DeleteTags
    DisableLogging
    EnableLogging
    ModifyCluster
    RebootCluster
    ResizeCluster
    RevokeSnapshotAccess

Route Tables:
    AssociateRouteTable
    CreateRoute
    CreateRouteTable
    DeleteRoute
    DeleteRouteTable
    DisassociateRouteTable
    ReplaceRoute
    ReplaceRouteTableAssociation
    
S3:
    CreateBucket
    DeleteBucket
    DeleteBucketEncryption
    DeleteBucketPolicy
    DeleteBucketPublicAccessBlock
    DeleteBucketTagging
    DeleteBucketWebsite
    PutBucketAcl
    PutBucketEncryption
    PutBucketLogging
    PutBucketPolicy
    PutBucketPublicAccessBlock
    PutBucketTagging
    PutBucketVersioning
    PutBucketWebsite
    
SNS Subscription:
    SetSubscriptionAttributes
    Unsubscribe   
    
SNS Topic:
    CreateTopic
    DeleteTopic
    SetTopicAttributes
    TagQueue
    UntagQueue
    
SSH Keypairs:
    CreateKeyPair
    DeleteKeyPair
    ImportKeyPair
    
Security Groups and Network ACLs:
    AuthorizeSecurityGroupEgress
    AuthorizeSecurityGroupIngress
    CreateNetworkAcl
    CreateNetworkAclEntry
    CreateSecurityGroup
    CreateTags
    DeleteNetworkAcl
    DeleteNetworkAclEntry
    DeleteSecurityGroup
    DeleteTags
    ReplaceNetworkAclAssociation
    ReplaceNetworkAclEntry
    RevokeSecurityGroupEgress
    RevokeSecurityGroupIngress
    UpdateSecurityGroupRuleDescriptionsEgress
    UpdateSecurityGroupRuleDescriptionsIngress
    
Service Access Key:
    CreateAccessKey
    DeleteAccessKey
    UpdateAccessKey
    
Subnets:
    CreateSubnet
    CreateTags
    DeleteSubnet
    DeleteTags
    
VPC Flow Logs:
    CreateFlowLogs
    DeleteFlowLogs
    
VPC Network Peer:
    AcceptVpcPeeringConnection
    CreateVpcPeeringConnection
    DeleteVpcPeeringConnection            
    RejectVpcPeeringConnection    

VPCs:
    AssociateDhcpOptions
    AssociateVpcCidrBlock
    CreateTags
    CreateVpc
    DeleteTags
    DeleteVpc
    DisassociateVpcCidrBlock

Updated about 3 hours ago

Event-Driven Harvesting (AWS)


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.