Operations

DivvyCloud has a three tier architecture that allows for flexible deployments to manage the scale and scope of small to large cloud footprints.

The application consists of multiple web servers, job schedulers, and workers. DivvyCloud uses Redis for coordination and MySQL as the persistance layer. Secondary web servers and job schedulers operate in stand-by mode. Secondary (and more) workers operate in multi-processing mode by default.

Application management and worker processes can be accessed using specific commands.

Commands

There are four supported commands available for running DivvyCloud processes and performing management tasks. All commands run within a docker container and require setting environment variables.

To view the options for each script, use the command with the flag <command> --help.

  • divvyinterfaceserver - web server process
  • divvycloudjobscheduler - scheduler process that schedules periodic harvesting and background tasks
  • divvycloudworker - worker process that receives jobs from scheduler
  • divvyadmin - administrative commands such as license registration, database backups, etc. Necessary only in special cases
  • divvycloud - commands for starting/stopping the whole suite. Deprecated, use orchestration tools like docker-compose.

The most important flag when using these commands is -n --nodaemon, which runs the process in the foreground. Processes should always be run in the foreground when using docker, because when the process exits, the docker container dies.

Another flag of note is divvycloudjobscheduler --do-db-upgrade, which performs database migrations for first time installations and upgrades. All other processes remain in a wait state until the appropriate database version condition is satisfied.

When using divvyadmin commands, you must connect to a single running docker container and run a command within it. To do so, run docker ps and use that id with the docker exec command. Then you are ready to use divvyadmin as follows.

docker ps

# CONTAINER ID        IMAGE                           ...
# abcdefg12345        divvycloud/divvycloud:latest    ...

docker exec -it abcdefg12345 /bin/bash

divvyadmin <command> <arg1> <argn...>

# Exit the container
exit

To check for what version of DivvyCloud you are running, you can check the top right hand corner on your dashboard.

_images/version.png

Worker Processes

Workers are divided among several queues. They are:

  • on-demand
  • harvester
  • harvester-long
  • processor

Queues can be selected via command option as a comma seperated list. For example: divvycloudworker -t on-demand, harvester.

You can see queue status in the System Administration section of DivvyCloud. When operating optimally, worker queues should be zero.

If you wish to assign a worker to a specific queue, remember that harvester-long jobs are expected to take a long time to process, so there must always be enough harvester workers to continue with normal harvesting if long harvesters are blocked.

Environment

Docker provides the ability to define environment variable values in a file, which is passed into the container via the --env-file flag. Alternatively, environment variable values can be set individually via the --env, -e flag. To read more about options when running a docker container, please reference the official docker documentation.

DivvyCloud requires values for the following environment variables. Place these in a file such as sample.env in the same directory as the docker-compose file.

# MySQL 5.7 Primary database
DIVVY_MYSQL_HOST=mysql
DIVVY_MYSQL_PORT=3306
DIVVY_MYSQL_USER=divvy
DIVVY_MYSQL_PASSWORD=divvy

# MySQL 5.7 Secure database
DIVVY_SECURE_MYSQL_HOST=mysql
DIVVY_SECURE_MYSQL_PORT=3306
DIVVY_SECURE_MYSQL_USER=divvy
DIVVY_SECURE_MYSQL_PASSWORD=divvy

# Redis
DIVVY_REDIS_HOST=redis
DIVVY_REDIS_PORT=6379


# Divvy Required - do not modify
VIRTUAL_ENV=/
DIVVY_MYSQL_DB_NAME=divvy
DIVVY_SECURE_MYSQL_DB_NAME=divvykeys

This env file can then be referenced in your docker-compose file like so:

services:
  scheduler:
    env_file:
      - ./sample.env

If you wish to run a single container via docker run, you can pass in your sample.env file using a command like the one below. Using docker run is not recommended for long running processes, but rather individual administrative commands.

docker run -it --env-file sample.env divvycloud/divvycloud:lastest <command> [options]

Finally, if you wish to run DivvyCloud using a Proxy Server, you can by updating your env file as documented in Server and Network Requirements documentation.

MySQL/Redis

We recommend using docker-compose to run DivvyCloud (as in the ../installation/testdrive instructions). Docker-compose assists in establishing the networking rules necessary for the containers to talk to each other. If you wish to run a docker container via docker run, however, we recommend modifying the example docker-compose file to contain only:

  1. DivvyCloud container for executing command
  2. MySQL
  3. Redis

And then modifying the container’s command to one of the options listed in the “Commands” documentation below.

services:
  <image_name>:
    command:
      - <command>
      - <arg 1>
      - <arg n...>

Troubleshooting

The three most likely cases of when you would need to troubleshoot your DivvyCloud installation are:

  • License refresh
  • MySQL and redis connections set improperly
  • NTP not running

DivvyCloud maintains a license for each installation centrally. If your license expires, you will need to contact DivvyCloud at support@divvycloud.com with your license fingerprint, which is given with the license expiration notification. The license fingerprint will look like AB:12:34:CD:5E:2B:74:EF:00:ab:67:12:77:36:D7:55:2B:4D:7C:48.

After you’ve notified DivvyCloud, you can wait for your license to refresh in approximately one hour or you can run the following commands on your DivvyCloud instance to accelerate the process. When DivvyCloud is running you can connect to a single running docker container and run a command within it.

First, run docker ps and copy the container-id for the Job Scheduler process.

Next, run the following commands:

docker ps

# CONTAINER ID        IMAGE                           ...
# abcdefg12345        divvycloud/divvycloud:latest    ...

docker exec -it abcdefg12345 /bin/bash

# Once connected to the container
divvyadmin refresh_license

# Exit the container
exit

The second most likely cause of trouble is improperly configured connection information between DivvyCloud’s instances and its MySQL and/or Redis instances. If these connections are not correct, then DivvyCloud cannot save its harvested data and/or the job scheduler cannot schedule jobs and the workers cannot get their assignments. To confirm your corrections are correct, return to your environment variables and make sure they match the appropriate settings.

Lastly, if you see Invalid Credentials on an account that you know has the correct cloud credentials, it could be that your system is not in sync.

When running the DivvyCloud software, it’s important timing is precise. When you send something that has a signature to it, it is timestamped. If there is more than 5 minutes of drift, the cloud provider will say it’s an expired signature and treat it as an authentication failure. The fix for this issue is to ensure NTP is running, and restart (update).

That will flip them back to valid credentials and you’ll start to see harvest information coming in again for those accounts.

Architecture

Below is a basic diagram that shows the overall architecture of a production environment:

_images/topology.png