Installing Remote Plugins

Expanded Plugin Documentation

InsightCloudSec supports extension via a comprehensive plugin system. Customers can add new content and workflows to support their unique requirements by writing their own extensions. This capability requires access to our source code and is only available to customers.

Reach out to us through the the Customer Support Portal for access to both the repo and the additional documentation and instructions surrounding plugins.

To ease plugin deployment, you can configure InsightCloudSec to download plugins from a remote location on system startup. The application can download these files from S3 or from arbitrary HTTP hosting, optionally with password protection.

Prerequisites

You must have InsightCloudSec plugins in .zip files. Each .zip archive should either contain exactly one plugin or a directory consisting of multiple plugins.

That .zip file must be hosted in:

  • an S3 bucket that your InsightCloudSec installation already has access to via the s3:GetObject permission
  • an S3 bucket that is either protected by basic auth or is open to the world
  • an arbitrary HTTP-accessible location, protected by basic auth or open to the world
  • access to the all EC2 instances hosting the Interface Server, Scheduler and Worker containers

Product name to be replaced

You may observe that some components, screen captures, or examples use our former product name, DivvyCloud. This doesn't affect the configuration or the product's functionality, and we will notify you as we replace these component names.

Interface

The current interface consists of a collection of environment variables, shown below:

NameDescriptionDefault
DIVVY_PLUGIN_REMOTEThis contains all remote plugin configuration settings as a JSON string. See data format below.\n\nNote: If this environment variable is set, the application will automatically use scheduler-coordinated plugin loading, unless the DIVVY_PLUGIN_LEGACY_LOADING or DIVVY_PLUGIN_DEV environments are set to True.{}
DIVVY_PLUGIN_DEPLOY_DIREnvironment variable which can be used to set the deployment directory when using remote plugin syncing.<SystemTmp>/divvy/deployed_plugins>

The DIVVY_REMOTE_PLUGIN environment variable is a list of JSON objects, each object representing a remote plugin store configuration described in this table:

NameRequiredDescriptionDefault
urlYesFull URL to a compressed plugin(s). The application will use the prefix on the URL to identify the specific remote resource as described below.N/A
multi_pluginsNoIf True, InsightCloudSec will extract the file in the url, and inspect each sub-folder as a unique plugin.False
usernameNoOptional username to use if required by the urlN/A
passwordNoOptional password if required to access the urlN/A

AWS S3 Bucket

If the prefix of the url starts with s3:// the application will support the following behavior:

  • If no username or password are specified, the application will assume that the InsightCloudSec scheduler is properly set up to support Instance AssumeRole to access that bucket.
  • If the username and password are specified, the application will attempt to authenticate with those credentials where username is the API Key and password is the Secret Key.

HTTP/HTTPS storage

If the url starts with http:// or https:// and a username/password are supplied, the application will use basic auth to attempt to authenticate with the endpoint. This means that the application will support fetching a private git repo.

Example

For example, you could specify the following:

bash
1
export DIVVY_PLUGIN_REMOTE='
2
[{"url":"s3://bucket_name/plugin.zip"},
3
{"url":"https://test/many_compressed_plugins.zip", "multi_plugins":true},
4
{"url": "http://github.com/example/Plugins/archive/master.zip", "multi_plugins":true, "username": "xxxxxx", "password": "yyyyyy"}]
5
'

Using this configuration:

  • fetches an archive of a single plugin from S3 using Instance AssumeRole ({"url":"s3://bucket_name/plugin.zip"}),
  • fetches, unauthenticated, a .zip file consisting of many plugins, from a non-GitHub endpoint ({"url":"https://test/many_compressed_plugins.zip", "multi_plugins":true}),
  • fetches, using basic authentication, an archive of many plugins hosted in a GitHub repo ({"url": "http://github.com/example/Plugins/archive/master.zip", "multi_plugins":true, "username": "xxxxxx", "password": "yyyyyy"}).

Steps to Implementing

This change should be added to prod.env on all EC2 instances.

  1. SSH to the EC2 instance, locate the directory where prod.env is saved, stop the docker container before updating:

    bash
    1
    docker-compose down
  2. Ensure the containers are stopped.

  3. Update/Modify the prod.env file:

    vim
    1
    vi prod.env
    2
    3
    # MySQL 5.7 Primary database
    4
    DIVVY_DB_HOST=mysql
    5
    DIVVY_DB_PORT=3306
    6
    DIVVY_DB_USERNAME=divvy
    7
    DIVVY_DB_PASSWORD=divvy
    8
    9
    # MySQL 5.7 Secure database
    10
    DIVVY_SECRET_DB_HOST=mysql
    11
    DIVVY_SECRET_DB_PORT=3306
    12
    DIVVY_SECRET_DB_USERNAME=divvy
    13
    DIVVY_SECRET_DB_PASSWORD=divvy
    14
    15
    # Redis
    16
    DIVVY_REDIS_HOST=redis
    17
    DIVVY_REDIS_PORT=6379
    18

    becomes

    vim
    1
    # MySQL 5.7 Primary database
    2
    DIVVY_DB_HOST=mysql
    3
    DIVVY_DB_PORT=3306
    4
    DIVVY_DB_USERNAME=divvy
    5
    DIVVY_DB_PASSWORD=divvy
    6
    7
    # MySQL 5.7 Secure database
    8
    DIVVY_SECRET_DB_HOST=mysql
    9
    DIVVY_SECRET_DB_PORT=3306
    10
    DIVVY_SECRET_DB_USERNAME=divvy
    11
    DIVVY_SECRET_DB_PASSWORD=divvy
    12
    13
    # Redis
    14
    DIVVY_REDIS_HOST=redis
    15
    DIVVY_REDIS_PORT=6379
    16
    DIVVY_PLUGIN_REMOTE='[{"url":"s3://bucket_name/plugin.zip"}]'
    17
  4. Start up the containers to apply the change:

    bash
    1
    docker-compose up
  5. Verify the variables have been set in the container:

    bash
    1
    docker ps
    2
    3
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    4
    0a512082abfe divvycloud/divvycloud:latest "/entrypoint.sh divv…" 35 minutes ago
    5
    6
    docker exec -it 0a512082abfe bash
    7
    8
    root@1b5bb7bebc37:/# env | grep DIVVY_PLUGIN_REMOTE
    9
    DIVVY_PLUGIN_REMOTE=[{"url":"s3://bucket_name/plugin.zip"}]

Advanced Configuration for Plugin Developers

The following environment variables are for use if you are developing plugins locally:

NameDescriptionDefault
DIVVY_PLUGIN_DEVControls if InsightCloudSec should be in Plugin Development Mode. Currently this has the same affect as setting the DIVVY_PLUGIN_LEGACY_LOADING variable.\n\nWill evaluate to True if the value is anything other than \"\" or \"False\".False
DIVVY_PLUGINSThis environment variable contains the path to locally installed plugins.plugins folder at the project/deploy root folder.