FinalDivvyCloud

18.7 Release Notes

7 months ago by Jenni Natiw

18.7.8 Maintenance Release Notes

Divvy Software Release Notice - 18.7.8 Maintenance Release (02/20/2019).

  • DV2019-361 Denote whether or not an AWS Aurora DB cluster has a read replica or not
  • DV2019-362 Add visibility into Lambda functions which are associated with application load balancers within AWS
  • DV2019-354 Handle empty/missing extensions when harvesting virtual machines from Microsoft Azure
  • DV2019-378 Add support for identifying Azure Storage Containers with/without encryption

18.7.6 Maintenance Release Notes

Divvy Software Release Notice - 18.7.6 Maintenance Release (02/13/2019).

  • Introduced fallback logic for AWS S3 tags, if we are unable to obtain the tags due to provider API issues
  • DV2019-301 Improved error handling for empty/missing snapshot listings on GCP
  • DV2019-322 Fixed false positives on hookpoint matches from index
  • Associate newly added AWS, Azure and GCP accounts with the default strategy for that cloud if a strategy ID is not supplied
  • DV2019-309 Converted AWS CloudFront harvesting to a global call to reduce the number of API calls made to the service
  • DV2019-313 Properly store AWS routes pointing to NAT/Transit gateways
  • DV2019-294 Added the ability to skip previously identified resources when posting resource data to a remote URL
  • Affix a permanent ResourceProperty divvy.creator which stores the user who provisioned AWS resources and can be used to enrich resource tagging. This only works for AWS customers who leverage EDH
  • DV2019-289 GCP Organization harvesting now no longer filters projects by parent ID which allows support for iterating Folders
  • Don't throw hookpoints on SNS and SQS policy documents which have not changed
  • DV2019-287 Leverage the OrphanedResourceCleanup processor to more efficiently removal resources/cloud accounts pending deletion
  • DV2019-284 Support additional DNS record types (NAPTR/CAA) within AWS Route53
  • DV2019-257 Properly store the OssBucketName that's associated with Alibaba Cloud API accounting resources

18.7.5 Maintenance Release Notes

Divvy Software Release Notice - 18.7.5 Maintenance Release (02/06/2019).

  • DV2019-248 Fixed a harvesting issue that prevented GCP Database Snapshots from harvesting for multi-node clusters
  • Improved equality checking for the IAM assume role policy document
  • DV2019-244 Added Request Timeout For Kubernetes
  • DV2019-167 Added the ability to sort Database Clusters by the lifecycle state
  • DV2019-243 Restored the ability to create Bots that run against DNS Zone resources
  • DV2019-221 Do not harvest AWS Organization information from AWS GovCloud or AWS China as it's not supported by the provider
  • DV2019-222 Added Namespace Search Filter for Kubernetes
  • DV2019-218 Do not include password in email for external auth users
  • DV2019-195 ECS Commands Not Showing
  • DV2019-266 Fixed an issue that prevented Bots from updating S3 IAM bucket policies
  • DV2019-266 Rephrased the ID field and made it a string input, not textarea
  • DIVVY-4488 Added NSX LoadBalancer Support
  • Added the ability for domain administrators to reset job scheduler stats within System Settings
  • Restored Instance CPU and Database connection harvesting in AWS
  • Fixed an edge case that prevented harvesting of Azure storage accountsd
  • Fixed an issue where missing images/job names were displaying in the Slowest Jobs panel

18.7.4 Maintenance Release Notes

Divvy Software Release Notice - 18.7.4 Maintenance Release (01/31/2019).

  • Restored multiprocessing on select resource types where it was enabled in 18.6
  • Added new Google Cloud Platform Filters to identify resources running in legacy/default networks
  • DV2019-204 Improved the tagging action, "Assign Tag to Resource" to prevent a possible loop from occurring
  • DV2019-194 Fixed an issue with Cloud Service Cost jobs failing to start
  • DV2019-191 Added Queue name to worker logs
  • DV2019-192: Restored the ability to create new custom Insights by setting the proper default value for badge scopes
  • Changed the cadence of the license harvester to run every 6 hours instead of every hour

18.7.3 Maintenance Release Notes

Divvy Software Release Notice - 18.7.3 Maintenance Release (01/28/2019).

  • Fixed an issue that resulted in GCP clouds being put into a paused state
  • Fixed an issue that would prevent scheduled events from firing under certain conditions
  • DV2019-178 - Fixed an edge case that prevented tag explorer configurations from being saved
  • DV2019-179 - Added wildcard and exclusion capabilities to the the API Key By User Name filter
  • DV2019-158 - Defaulted to newly created Insight to leverage OR vs AND when multiple badges are associated with an Insight
  • DV2019-168 - Removed duplicate deletion protection column from the Database Clusters view
  • DV2019-167 - Fixed DatabaseCluster region sorting
  • DV2019-172 - AWS EFS visibility into eu-west-2 (London), Workspace visibility into AWS GovCloud us-west-1, and Kinesis Firehose visibility into AWS China (cn-north-1)
  • DV2019-159 - Immediately enqueued jobs for a newly added cloud
  • DIVVY-4557 - Expanded the Resource Recently Modified filter to support all resource types
  • DV2019-162- Fixed an incorrect version formatting of backoffice api url
  • DV2019-175 - Fixed an issue were we did not actually remove dead keys from sets

Upgrade Prerequisites

Before upgrading your DivvyCloud installation, please take a moment to view our Administrator and Developer Notes at the bottom of this page.

Setup Harvest Strategy after you Upgrade to 18.7

It's very important that you set up your harvesting strategies after you upgrade to 18.7. More information on configuring Harvesting Strategies can be found here.

Contact support@divvycloud.com for questions.

Release Highlights

New Job Scheduler

18.7 introduces a new scheduler with major performance enhancements on the back end. The scheduler is now much faster and more stable, especially at scale. Start up time is a fraction of what it was prior to 18.7; any refresh or loading times via either the UI or API are also reduced. Not only is the scheduler faster, it also allows for a very low CPU footprint. After deploying 18.7, customers can check hardware to either downsize extra large instances, or even eliminate some worker nodes.

Pack Compliance Detail by Account

Accessible by clicking on an account card in the Cloud Compliance view, the Pack Compliance page displays: a compliance time-series graph, a listing of compliance per every rule in the pack, and a breakdown of noncompliance by resource type.

Cloud Organizations - Account Sync

A new tab, 'Organizations', has been introduced to the Clouds page. Within the page, a cloud organization resource can be added to the system to enable automatic addition of all cloud accounts under the organization into DivvyCloud system. Only Google Cloud Platform organizations will be supported in this release. We will be expanding this feature to AWS Organizations in early 2019.

Harvesting Strategy & Overrides

Harvesting strategies and overrides are new items introduced in this release which will help customize the cadence of resource harvesters at a granular level. There will be a default strategy for AWS, Google, and Azure, and any existing cloud accounts will be assigned the respective default strategy. Create new strategies or configure overrides for existing strategies for the suitable harvesting needs. Admins can access the Harvesting Strategy page via the main navigation panel. More information on configuring Harvesting Strategies can be found here.

Domain Viewer - Read Only Admin

There's now a new domain user classification, Domain Viewer, which allows a user to be given full read-only access to the entire installation; however, users of this type cannot take any lifecycle operations on cloud resources, create Insights, Bots, or any other administrative function within the tool. This feature is especially useful for customers running multiple organizations.

Additional Cloud Support/Enhancements

Amazon Web Services

* Support for Container Registries and Images
* Support for Account Level S3 Bucket Access Controls
* Support for EC2 instance hibernation
* Support for new A1 and C5n instance types
* Event driven harvesting (EDH) support for VPC Flow Logs, Dedicated Hosts, Network Peers, Memcache and Elasticsearch Instances, and RDS Aurora clusters.
* Enhanced visibility and lifecycle support for RDS Aurora Clusters
* Storage of the resource ARN across all resource types
* Support for AWS Cloudwatch Logs
* Added support for us-gov-east-1 region
* Add visibility into whether or not an instance is a spot instance
* Support for SageMaker Notebooks
* Add support for tags for IAM users/roles
* Add ability to suspend and resume processes for Autoscaling Groups
* Support for the new Stockholm region (eu-north-1)

Google Cloud Platform

* Support for walking a GCP Organization and auto-adding all projects
* Support for Autoscaling Groups
* Support for Cloud DNS
* Add the ability to start/stop SQL databases

Microsoft Azure

* Support for Container Registries
* Support for harvesting of IAM Service Principals with role assignments
* Support for harvesting of App Service Plans and Apps
* Support for harvesting Azure Functions

VMware

* Support for creating a new VM from an existing instance snapshot

General Enhancements

  • Job Backlog data can now be exported to CloudWatch, configured via SystemSettings table.
  • Event driven harvesting producer configuration can be updated within the tool.
  • Added cross account inspection for Service Encryption Keys and Elasticsearch Instances.
  • Added pagination support for KMS encryption keys, enabling visibility into regions with over 1,000 keys.
  • Split Database Instances and Database Clusters into two separate resource types for more clarity.
  • Added encryption type specification support to the Storage Containers With Default Encryption resource filter.
  • Converted divvy. badge prefix to system. to add clarity that these are system level badges which cannot be modified/removed.
  • Added ability to track the state of Internet Gateways.

Event Driven Harvesting (EDH)

The following AWS events have been added:

    s3control:DeleteAccountPublicAccessBlock
    s3control:PutAccountPublicAccessBlock

Highlighted Filters

We released over 100 filters in 18.7 and we wanted to highlight a few that we thought were interesting (see below).

To view a complete list all the new filters in 18.7, please check out the Filters Page in your DivvyCloud account.

Cloud Account Without Bucket Public Policies Enabled (AWS):

Identifies AWS accounts without specified bucket policy option ''enabled’.

DNS Zone Without DNS Security (GCP):

Match DNS zones which have DNSSEC (DNS Security) disabled.

Kubernetes CIS:

Over 40 new filters have been added for Kubernetes Cloud Accounts.

Service Detector Does Not Have Specified Master Account(AWS):

Identify service detectors that do not have any of the supplied accounts as their master accounts. The master account invites other accounts to become associated with it. Those accounts are called member accounts. A master account can view and manage its member accounts.

Cloud Role Trust Policy Contains Wildcard Name(AWS):

Match cloud roles with a trust relationship which contains access 'from a wildcard role name.

Storage Container With IAM Policy Statement(AWS):

Identify storage containers with an IAM policy that contain the supplied statement ID.

Log Group Without Encryption At Rest(AWS):

Identify log groups which do not have encryption enabled.

Serverless Function Allows Anonymous Access (Azure):

Identify Serverless Functions which have an HTTP binding that does not require authentication.

Instance Is Spot Instance

Identify instances which are of type Spot Instance.

Instance Is Not Spot Instance

Identify instances which are not of type Spot Instance.

Instance With Detailed Monitoring Enabled

Identify instances for which detailed monitoring is enabled. In clouds such as Amazon Web Services, detailed monitoring has an additional cost, so this filter can help identify opportunities for cost savings.

Instance With Detailed Monitoring Disabled

Identify instances for which detailed monitoring is disabled. Detailed monitoring can be important for production systems where additional monitoring capabilities are required.

Instance Without Required Extensions

Identify instances which are missing one or more required extensions. The supplied extension(s) must be present on the target instances and provisioned in a successful state.

ML Instance Lifecycle State

Identify ML instances in a particular lifecycle state.

ML Instance Direct Access

Identify ML instances that are directly accessible from the Internet.

ML Instance Not Associated With Network

Identify ML instances that are not part of a subnet.

App With Remote Debugging State

Match web apps with the selected remote debugging state.

App With Web Sockets State

Match web apps with the selected web sockets state.

App With FTP Configuration State

Match web apps with the selected FTP configuration state.

App With HTTPS Configuration State

Match web apps with the selected HTTPS configuration state.

Serverless Function Allows Anonymous Access

Identify Serverless Functions which have an HTTP bindings that does not require authentication. This only applies to Microsoft Azure Functions.

Internet Gateway Attached

Match Internet Gateways which are associated with a parent network

NAT Gateway Lifecycle State

Match NAT Gateways which are in the specified lifecycle state(s).

New/Improved Actions

Suspend AWS Autoscaling Group Processes

Suspend automatic scaling processes for an autoscaling group. You can specify certain processes or do all at once.

Resume AWS Autoscaling Group Processes

Resume automatic scaling processes for an autoscaling group. You can specify certain processes or do all at once.

Delete AWS Database Cluster

Execute a delete lifecycle command against an AWS RDS Cluster. Note that this lifecycle command will permanently destroy the master write node as well as all read-only replicas.

Start AWS Database Cluster

Execute a start lifecycle command against an AWS RDS Cluster. Note that this lifecycle command will start the master write node as well as all read-only replicas.

Stop AWS Database Cluster

Execute a stop lifecycle command against an AWS RDS Cluster. Note that this lifecycle command will stop the master write node as well as all read-only replicas.

Delete GCP User

Remove all permissions for a particular GCP User from a project. This action can be taken manually or via a Bot.

Delete GCP Group

Remove all permissions for a particular GCP Group from a project. This action can be taken manually or via a Bot.

Delete GCP Role

Remove all permissions for a particular GCP Role (Service Account) from a project. This action can be taken manually or via a Bot.

Enable Detailed Monitoring

Enable detailed monitoring for AWS compute instances. When enabled, performance data is available in 1-minute periods for an additional cost. For those instances for which you've enabled detailed monitoring, you can also get aggregated data across groups of similar instances.

Disable Detailed Monitoring

Disable detailed monitoring for AWS compute instances. When disabled, performance data is available in 5-minute periods without additional cost. Disable monitoring to save money on those instances which do not require additional granularity in performance data.

Bug Fixes

  • Fixed a visibility bug which was preventing plugins from displaying their load status in the UI (DIVVY-4299)
  • Fixed an edge case that prevented sorting within the Tag Explorer (DIVVY-3869)
  • Insight CSV download now takes cloud filtering into consideration (DIVVY-3939)
  • Delivery Streams no longer trigger modification hookpoints on every single harvest (DIVVY-4464)

Developer/Administrator Notes

IMPORTANT

Amazon Web Service

For Amazon Web Service customers, the role/user policy associated with each connected account will need to be adjusted to include the permissions below. Without these permissions, visibility into the newly supported AWS services will not be possible.

iam:ListInstanceProfiles
sagemaker:ListNotebookInstances
sagemaker:DescribeNotebookInstance
sagemaker:ListTags
s3:GetPublicAccessBlock
s3control:GetAccountPublicAccessBlock
autoscaling:SuspendProcesses
autoscaling:ResumeProcesses

For those interested in modifying tags for AWS IAM users and roles, the
following permissions will need to be included in your policy.

iam:TagUser
iam:TagRole
iam:UntagUser
iam:UntagRole
ecr:DescribeRepositories
ecr:DescribeImages

IMPORTANT

Microsoft Azure

For Microsoft Azure customers using custom roles, the role definitions associated with each connected account will need to be adjusted to include the permissions below. Without
these permissions, visibility into newly supported Microsoft Azure services will not be possible.

// For the read-only Standard User role
Microsoft.ContainerRegistry/registries/read
Microsoft.ContainerService/managedClusters/read
Microsoft.Storage/storageAccounts/read
Microsoft.Web/serverfarms/read
Microsoft.Web/sites/*/read
Microsoft.Web/sites/config/list/Action

// For the Power User role
Microsoft.ContainerRegistry/*
Microsoft.ContainerService/*
Microsoft.Storage/*
Microsoft.Web/*

Google Analytics

DivvyCloud v18.7 includes support for Google Analytics in order for us to improve navigation and the overall customer experience.