FinalDivvyCloud

Legacy Deployment

This documentation focuses on prior installations of DivvyCloud that do not employ Docker for deployment. If you are installing an older version of DivvyCloud, please feel free to reach out to us at support@divvycloud.com to help you upgrade.

Provision Resources

See Server and Network Recommendations to provision the recommended resources for DivvyCloud’s front end and back end services.

Configure MySQL

Configure MySQL with the required databases and access permissions by using the code snippet below. Note that you will need to replace <DivvyCloud.ip.range.here> with the IP address block which will be assigned to your DivvyCloud nodes.

CREATE DATABASE divvy;
CREATE DATABASE divvykeys;
GRANT ALL PRIVILEGES on divvy.* to 'divvy'@'<DivvyCloud.ip.range.here>' IDENTIFIED BY 'divvy';
GRANT ALL PRIVILEGES on divvykeys.* to 'divvy'@'<DivvyCloud.ip.range.here>' IDENTIFIED BY 'divvy';
GRANT RELOAD ON *.* TO 'divvy'@'<DivvyCloud.ip.range.here>' IDENTIFIED BY 'divvy';
FLUSH PRIVILEGES;

Install System Prerequisites

On each instance, you need to install the relevant system prerequisites.

For Ubuntu Distributions:

sudo apt-get update
sudo apt-get upgrade -yq
sudo apt-get -y install libssl-dev libxml2-dev libxslt1-dev python-dev gcc libldap2-dev libsasl2-dev libssl-dev libffi-dev libyaml-dev lib32z1-dev zlib1g-dev libmysqlclient-dev libmysqlclient-dev git make

For CentOS Distributions:

sudo yum -y install openssl-devel python-devel gcc libxml2-devel libxslt-devel libffi-devel openldap-devel libyaml-devel libmysqlclient-devel git

Configure Python on DivvyCloud Nodes

To ensure you are running Python 2.7.12, we leverage pyenv to avoid interfering with the version of Python that ships with your OS distribution.

The snippet below downloads the git executable, clones the pyenv repository, and sets up a dedicated virtual environment. We recommend doing this as a dedicated user such as divvy.

git clone https://github.com/yyuu/pyenv.git ~/.pyenv
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
exec $SHELL
source ~/.bash_profile
pyenv install 2.7.12
pyenv local 2.7.12

Once you’ve completed the installation, if you type python your prompt should look similar to the output below:

[divvy@ip-172-31-3-245 ~]$ python
Python 2.7.12 (default, May  8 2017, 14:29:12)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Download & Run DivvyCloud Packages

Now you are ready to download and install DivvyCloud. Again, we recommend doing this as the dedicated user such as divvy and we recommend using /home/divvy/divvycloud as your installation directory. Run the commands below on each node from the installation directory.

pip install virtualenv
virtualenv divvycloud
cd divvycloud ; source bin/activate
export INSTALL_URL='http://packages.divvycloud.com/packages'
easy_install --upgrade -f ${INSTALL_URL} divvycloud divvycloudprocessmanager

Complete the ~/.bash_profile

To ensure that MySQL and Redis are pointing to the correct endpoints, whether you're running them locally or remotely, you'll need to set the bash_profile on the operating system level by editing ~/.bash_profile:

# MySQL 5.7 Primary database
export DIVVY_DB_HOST=YOURMYSQLHOST-ENDPOINT
export DIVVY_DB_PORT=3306
export DIVVY_DB_USERNAME=xxxxxxx
export DIVVY_DB_PASSWORD=yyyyyyy

# MySQL 5.7 Secure database
export DIVVY_SECRET_DB_HOST=YOURMYSQLHOST-ENDPOINT
export DIVVY_SECRET_DB_PORT=3306
export DIVVY_SECRET_DB_USERNAME=xxxxxxxx
export DIVVY_SECRET_DB_PASSWORD=yyyyyyyy

# Redis
export DIVVY_REDIS_HOST=YOURREDIS-ENDPOINT
export DIVVY_REDIS_PORT=6379

# Divvy Required - do not modify
export VIRTUAL_ENV=/
export DIVVY_DB_NAME=divvy
export DIVVY_SECRET_DB_NAME=divvykeys

You can validate your bash_profile by cat ~/.bash_profile

(divvycloud) divvycloud@ip-172-31-8-240:~/divvycloud$ cat ~/.bash_profile
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
# MySQL 5.7 Primary database
export DIVVY_MYSQL_HOST=localhost
export DIVVY_MYSQL_PORT=3306
export DIVVY_MYSQL_USER=divvy
export DIVVY_MYSQL_PASSWORD=divvy

# MySQL 5.7 Secure database
export DIVVY_SECURE_MYSQL_HOST=localhost
export DIVVY_SECURE_MYSQL_PORT=3306
export DIVVY_SECURE_MYSQL_USER=divvy
export DIVVY_SECURE_MYSQL_PASSWORD=divvy

# Redis
export DIVVY_REDIS_HOST=localhost
export DIVVY_REDIS_PORT=6379

# Divvy Required - do not modify
export VIRTUAL_ENV=/
export DIVVY_MYSQL_DB_NAME=divvy
export DIVVY_SECURE_MYSQL_DB_NAME=divvykeys

Run Backend Migration Scripts

The commands below need to be run only one time on a single system from the installation directory. They will run the necessary migration scripts.

source bin/activate
divvyadmin upgrade-database

Register License

On a single node, you need to register a new license. Registering will grant you a 30-day license. Please contact us at support@divvycloud.com to extend it.

divvyadmin register --company_name "Company Name Here" --contact_email "contactemail@acmecorp.com" --contact_name "Contact Name"

Start The Suite

You should now be ready to go. You can start the suite up by creating and running the following script from the installation directory:

#!/bin/bash

# Read our env variables
source ~/.bash_profile

echo "Starting InterfaceServer"
divvycloud webserver > /dev/null 2>&1 &
sleep 10
echo "Starting Scheduler"
divvycloud scheduler --db-upgrade > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #1"
divvycloud worker > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #2"
divvycloud worker > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #3"
divvycloud worker > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #4"
divvycloud worker > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #5"
divvycloud worker > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #6"
divvycloud worker > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #7"
divvycloud worker > /dev/null 2>&1 &
sleep 10
echo "Starting Worker #8"
divvycloud worker > /dev/null 2>&1 &
echo "Done!"

If you receive permissions errors when running this script, chmod +x start.sh

source bin/activate
./start.sh

to stop:

source bin/activate
./stop.sh
#!/bin/bash

kill $(ps aux | grep 'divvycloud' | grep -v grep | awk '{print $2}')

Scaling Up

Should you need to scale the suite, you can add instances and then assign specific processes to those instances by modifying your `start.sh' file to add more workers.

webserver => Run only the interface (web) server
worker => Run only the worker/harvest processes
scheduler => Run only the job scheduler
upgrade-database => Apply database migrations to divvy and divvykeys

Legacy Upgrade Instructions

For 18.6 to 18.7 upgrades use this guide

Each of these commands requires the virtual environment to be activated. Do so by running this command and then stop DivvyCloud.

cd </home/divvy/divvycloud> ; source bin/activate
./stop.sh

For simplicity, export the package server url.

export INSTALL_URL='http://packages.divvycloud.com/packages'

For the latest version run this command:

easy_install --upgrade -f ${INSTALL_URL} divvycloud

If you need to identify a specific version of DivvyCloud to which you are upgrading. You can do that by browsing our repository at http://packages.divvycloud.com/simple/divvycloud to see the egg files for DivvyCloud with the most recent release at the bottom.

export DIVVY_VERSION='<major.minor.point | e.g., 17.03.33>'
easy_install --upgrade -f ${INSTALL_URL} divvycloud==${DIVVY_VERSION} divvycloudprocessmanager==${DIVVY_VERSION}

Restart DivvyCloud

./start.sh

To check if divvycloud is running:

ps aux | grep divvycloud
(divvycloud) divvycloud@ip-172-31-12-121:~/divvycloud$ ps aux | grep divvycloud
ubuntu   16293  6.0  7.0 810548 286420 pts/0   Sl   21:34   0:04 /home/ubuntu/divvycloud/bin/python2.7 /home/ubuntu/divvycloud/bin/divvycloud webserver
ubuntu   16299  6.1  6.8 869072 276524 pts/0   Sl   21:34   0:04 /home/ubuntu/divvycloud/bin/python2.7 /home/ubuntu/divvycloud/bin/divvycloud scheduler --db-upgrade
ubuntu   16307 12.7  7.0 435660 283260 pts/0   S    21:34   0:09 /home/ubuntu/divvycloud/bin/python2.7 /home/ubuntu/divvycloud/bin/divvycloud worker
ubuntu   16320  6.3  6.8 431244 278848 pts/0   S    21:34   0:04 /home/ubuntu/divvycloud/bin/python2.7 /home/ubuntu/divvycloud/bin/divvycloud worker
ubuntu   16332  6.6  6.8 430580 278228 pts/0   S    21:34   0:04 /home/ubuntu/divvycloud/bin/python2.7 /home/ubuntu/divvycloud/bin/divvycloud worker
ubuntu   16340  6.9  6.8 430572 278488 pts/0   S    21:34   0:04 /home/ubuntu/divvycloud/bin/python2.7 /home/ubuntu/divvycloud/bin/divvycloud worker
ubuntu   16348  7.1  6.8 430576 278316 pts/0   S    21:34   0:04 /home/ubuntu/divvycloud/bin/python2.7 /home/ubuntu/divvycloud/bin/divvycloud worker
ubuntu   16354  7.3  6.8 430596 278356 pts/0   S    21:34   0:04 /home/ubuntu/divvycloud/bin/python2.7 /home/ubuntu/divvycloud/bin/divvycloud worker
ubuntu   16364  7.5  6.8 430576 278700 pts/0   S    21:34   0:04 /home/ubuntu/divvycloud/bin/python2.7 /home/ubuntu/divvycloud/bin/divvycloud worker
ubuntu   16370  7.7  6.8 430576 278484 pts/0   S    21:34   0:04 /home/ubuntu/divvycloud/bin/python2.7 /home/ubuntu/divvycloud/bin/divvycloud worker
ubuntu   16397  0.0  0.0  12944   972 pts/0    S+   21:35   0:00 grep divvycloud

If you receive an error after upgrading

destroy the virtualenv and recreate it:

rm -rf divvycloud

virtualenv divvycloud

Now reinstall the divvycloud package:

cd divvycloud
source bin/activate
export INSTALL_URL='https://packages.divvycloud.com/packages'
pip install -f $INSTALL_URL divvycloud