Installing Remote Plugins

📘
Expanded Plugin Documentation
InsightCloudSec supports extension via a comprehensive plugin system. Customers can add new content and workflows to support their unique requirements by writing their own extensions. This capability requires access to our source code and is only available to customers.
Reach out to us through the the Customer Support Portal for access to both the repo and the additional documentation and instructions surrounding plugins.

To ease plugin deployment, you can configure InsightCloudSec to download plugins from a remote location on system startup. The application can download these files from S3 or from arbitrary HTTP hosting, optionally with password protection.

Prerequisites

You must have InsightCloudSec plugins in .zip files. Each .zip archive should either contain exactly one plugin or a directory consisting of multiple plugins.

That .zip file must be hosted in:

an S3 bucket that your InsightCloudSec installation already has access to via the s3:GetObject permission
an S3 bucket that is either protected by basic auth or is open to the world
an arbitrary HTTP-accessible location, protected by basic auth or open to the world
access to the all EC2 instances hosting the Interface Server, Scheduler and Worker containers

📘
DivvyCloud vs. InsightCloudSec
Note that examples, variables, and database components may still refer to DivvyCloud vs. InsightCloudSec - the functionality is the same.

Interface

The current interface consists of a collection of environment variables, shown below:

Name	Description	Default
`DIVVY_PLUGIN_REMOTE`	This contains all remote plugin configuration settings as a JSON string. See data format below. Note: If this environment variable is set, the application will automatically use scheduler-coordinated plugin loading, unless the `DIVVY_PLUGIN_LEGACY_LOADING` or `DIVVY_PLUGIN_DEV` environments are set to `True`.	`{}`
`DIVVY_PLUGIN_DEPLOY_DIR`	Environment variable which can be used to set the deployment directory when using remote plugin syncing.	`<SystemTmp>/divvy/deployed_plugins>`

The DIVVY_REMOTE_PLUGIN environment variable is a list of JSON objects, each object representing a remote plugin store configuration described in this table:

Name	Required	Description	Default
`url`	Yes	Full URL to a compressed plugin(s). The application will use the prefix on the URL to identify the specific remote resource as described below.	N/A
`multi_plugins`	No	If True, InsightCloudSec will extract the file in the url, and inspect each sub-folder as a unique plugin.	False
`username`	No	Optional username to use if required by the url	N/A
`password`	No	Optional password if required to access the url	N/A

AWS S3 Bucket:

If the prefix of the url starts with s3:// the application will support the following behavior.

If no username or password are specified, the application will assume that the InsightCloudSec scheduler is properly set up to support Instance AssumeRole to access that bucket.

If the username and password are specified, the application will attempt to authenticate with those credentials where username is the API Key and password is the Secret Key.

HTTP/HTTPS storage

If the url starts with http:// or https:// and a username/password are supplied, the application will use basic auth to attempt to authenticate with the endpoint. This means that the application will support fetching a private git repo.

Example

For example, you could specify the following:

export DIVVY_PLUGIN_REMOTE='
[{"url":"s3://bucket_name/plugin.zip"},
{"url":"https://test/many_compressed_plugins.zip", "multi_plugins":true}, 
{"url": "http://github.com/example/Plugins/archive/master.zip", "multi_plugins":true, "username": "xxxxxx", "password": "yyyyyy"}]
'

Use this configuration:

fetches an archive of a single plugin from S3 using Instance AssumeRole ({"url":"s3://bucket_name/plugin.zip"}),
fetches, unauthenticated, a zipfile consisting of many plugins, from a non-GitHub endpoint ({"url":"https://test/many_compressed_plugins.zip", "multi_plugins":true}),
fetches, using basic authentication, an archive of many plugins hosted in a GitHub repo ({"url": "http://github.com/example/Plugins/archive/master.zip", "multi_plugins":true, "username": "xxxxxx", "password": "yyyyyy"}).

Steps to Implementing

Note: This change should be added to prod.env on all EC2 instances

SSH to the EC2 instance, locate the directory where prod.env is saved, stop the docker container before updating:

docker-compose down

Ensure the containers are stopped.
Update/Modify the prod.env file:

vi prod.env

# MySQL 5.7 Primary database
DIVVY_DB_HOST=mysql
DIVVY_DB_PORT=3306
DIVVY_DB_USERNAME=divvy
DIVVY_DB_PASSWORD=divvy

# MySQL 5.7 Secure database
DIVVY_SECRET_DB_HOST=mysql
DIVVY_SECRET_DB_PORT=3306
DIVVY_SECRET_DB_USERNAME=divvy
DIVVY_SECRET_DB_PASSWORD=divvy

# Redis
DIVVY_REDIS_HOST=redis
DIVVY_REDIS_PORT=6379

becomes

# MySQL 5.7 Primary database
DIVVY_DB_HOST=mysql
DIVVY_DB_PORT=3306
DIVVY_DB_USERNAME=divvy
DIVVY_DB_PASSWORD=divvy

# MySQL 5.7 Secure database
DIVVY_SECRET_DB_HOST=mysql
DIVVY_SECRET_DB_PORT=3306
DIVVY_SECRET_DB_USERNAME=divvy
DIVVY_SECRET_DB_PASSWORD=divvy

# Redis
DIVVY_REDIS_HOST=redis
DIVVY_REDIS_PORT=6379
DIVVY_PLUGIN_REMOTE='[{"url":"s3://bucket_name/plugin.zip"}]'

Start up the containers to apply the change:

docker-compose up

Verify the variables have been set in the container:

docker ps

CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS                         PORTS                     NAMES
0a512082abfe        divvycloud/divvycloud:latest   "/entrypoint.sh divv…"   35 minutes ago 

docker exec -it 0a512082abfe bash

root@1b5bb7bebc37:/# env | grep DIVVY_PLUGIN_REMOTE
DIVVY_PLUGIN_REMOTE=[{"url":"s3://bucket_name/plugin.zip"}]

Advanced Configuration for Plugin Developers

The following environment variables are for use if you are developing plugins locally:

Name	Description	Default
`DIVVY_PLUGIN_DEV`	Controls if InsightCloudSec should be in Plugin Development Mode. Currently this has the same affect as setting the `DIVVY_PLUGIN_LEGACY_LOADING` variable. Will evaluate to `True` if the value is anything other than `""` or `"False"`.	False
`DIVVY_PLUGINS`	This environment variable contains the path to “locally” installed plugins.	“plugins” folder at the project/deploy root folder.