DivvyCloud has a three tier architecture that allows for flexible deployments to manage the scale and scope of small to large cloud footprints.
The application consists of multiple web servers, job schedulers, and workers. DivvyCloud uses Redis for coordination and MySQL as the persistance layer. Secondary web servers and job schedulers operate in stand-by mode. Secondary (and more) workers operate in multi-processing mode by default.
Application management and worker processes can be accessed using specific commands.
There are four supported commands available for running DivvyCloud processes and performing management tasks. All commands run within a docker container and require setting environment variables.
To view the options for each script, use the command with the flag
divvyinterfaceserver- web server process
divvycloudjobscheduler- scheduler process that schedules periodic harvesting and background tasks
divvycloudworker- worker process that receives jobs from scheduler
divvyadmin- administrative commands such as license registration, database backups, etc. Necessary only in special cases
divvycloud- commands for starting/stopping the whole suite. Deprecated, use orchestration tools like docker-compose.
The most important flag when using these commands is
-n --nodaemon, which runs the process in the foreground. Processes should always be run in the foreground when using docker, because when the process exits, the docker container dies.
Another flag of note is
divvycloudjobscheduler --do-db-upgrade, which performs database migrations for first time installations and upgrades. All other processes remain in a wait state until the appropriate database version condition is satisfied.
divvyadmin commands, you must connect to a single running docker container and run a command within it. To do so, run
docker ps and use that id with the docker exec command. Then you are ready to use
divvyadmin as follows.
docker ps # CONTAINER ID IMAGE ... # abcdefg12345 divvycloud/divvycloud:latest ... docker exec -it abcdefg12345 /bin/bash divvyadmin <command> <arg1> <argn...> # Exit the container exit
To check for what version of DivvyCloud you are running, you can check the top right hand corner on your dashboard.
Workers are divided among several queues. They are:
Queues can be selected via command option as a comma seperated list. For example:
divvycloudworker -t on-demand, harvester.
You can see queue status in the System Administration section of DivvyCloud. When operating optimally, worker queues should be zero.
If you wish to assign a worker to a specific queue, remember that
harvester-long jobs are expected to take a long time to process, so there must always be enough
harvester workers to continue with normal harvesting if long harvesters are blocked.
Docker provides the ability to define environment variable values in a file, which is passed into the container via the
--env-file flag. Alternatively, environment variable values can be set individually via the
--env, -e flag. To read more about options when running a docker container, please reference the official docker documentation.
DivvyCloud requires values for the following environment variables. Place these in a file such as sample.env in the same directory as the docker-compose file.
DivvyCloud requires values for the following environment variables. Place these in a file such as
sample.env in the same directory as the docker-compose file.
# MySQL 5.7 Primary database DIVVY_MYSQL_HOST=mysql DIVVY_MYSQL_PORT=3306 DIVVY_MYSQL_USER=divvy DIVVY_MYSQL_PASSWORD=divvy # MySQL 5.7 Secure database DIVVY_SECURE_MYSQL_HOST=mysql DIVVY_SECURE_MYSQL_PORT=3306 DIVVY_SECURE_MYSQL_USER=divvy DIVVY_SECURE_MYSQL_PASSWORD=divvy # Redis DIVVY_REDIS_HOST=redis DIVVY_REDIS_PORT=6379 # Divvy Required - do not modify VIRTUAL_ENV=/ DIVVY_MYSQL_DB_NAME=divvy DIVVY_SECURE_MYSQL_DB_NAME=divvykeys
This env file can then be referenced in your docker-compose file like so:
services: scheduler: env_file: - ./sample.env
If you wish to run a single container via
docker run, you can pass in your
sample.env file using a command like the one below. Using
docker run is not recommended for long running processes, but rather individual administrative commands.
docker run -it --env-file sample.env divvycloud/divvycloud:lastest <command> [options]
Finally, if you wish to run DivvyCloud using a Proxy Server, you can by updating your env file as documented in Server and Network Requirements documentation.
We recommend using docker-compose to run DivvyCloud (as in the ../installation/testdrive instructions). Docker-compose assists in establishing the networking rules necessary for the containers to talk to each other. If you wish to run a docker container via
docker run, however, we recommend modifying the example docker-compose file to contain only:
- DivvyCloud container for executing command
And then modifying the container’s command to one of the options listed in the “Commands” documentation below.
services: <image_name>: command: - <command> - <arg 1> - <arg n...>
The three most likely cases of when you would need to troubleshoot your DivvyCloud installation are:
- License refresh
- MySQL and redis connections set improperly
- NTP not running
DivvyCloud maintains a license for each installation centrally. If your license expires, you will need to contact DivvyCloud at email@example.com with your license fingerprint, which is given with the license expiration notification. The license fingerprint will look like
After you’ve notified DivvyCloud, you can wait for your license to refresh in approximately one hour or you can run the following commands on your DivvyCloud instance to accelerate the process. When DivvyCloud is running you can connect to a single running docker container and run a command within it.
docker ps and copy the container-id for the Job Scheduler process.
Next, run the following commands:
docker ps # CONTAINER ID IMAGE ... # abcdefg12345 divvycloud/divvycloud:latest ... docker exec -it abcdefg12345 /bin/bash # Once connected to the container divvyadmin refresh_license # Exit the container exit
The second most likely cause of trouble is improperly configured connection information between DivvyCloud’s instances and its MySQL and/or Redis instances. If these connections are not correct, then DivvyCloud cannot save its harvested data and/or the job scheduler cannot schedule jobs and the workers cannot get their assignments. To confirm your corrections are correct, return to your environment variables and make sure they match the appropriate settings.
Lastly, if you see Invalid Credentials on an account that you know has the correct cloud credentials, it could be that your system is not in sync.
When running the DivvyCloud software, it’s important timing is precise. When you send something that has a signature to it, it is timestamped. If there is more than 5 minutes of drift, the cloud provider will say it’s an expired signature and treat it as an authentication failure. The fix for this issue is to ensure NTP is running, and restart (update).
That will flip them back to valid credentials and you’ll start to see harvest information coming in again for those accounts.
Below is a basic diagram that shows the overall architecture of a production environment: