CommCare Cloud

About this changelog

The following changes to commcare-cloud that require your attention, newest first.

Changelog

2019-08-30 Upgrade Sentry

The Sentry SDK used by CommCare HQ has been updated and along with it we have updated the configuration parameters.

2019-08-23 Add deploy Command

In order to provide a consistent user interface while making underlying changes, we are replacing the commcare-cloud <env> fab deploy command with a more concise commcare-cloud <env> deploy command.

2019-08-23 Removing support for Riak CS

We are removing support for deploying Riak CS clusters in commcare-cloud

2019-08-23 Fix python3 virutalenvs

This fixes a bug with how python3 virtualenvs were created by ansible. This fix needs to be applied to any machine which has a python3 virtualenv that was created by commcare-cloud.

The fix is also safe to run on all CommCare hosts.

2019-08-21 Move remaining management commands explicitly

This change requires editing app-processes.yml to add some of processes to the management_comamnds section

2019-08-20 Rename management commands explicitly

This change requires editing app-processes.yml to rename some of the processes in the management_comamnds section

2019-07-17 Define management commands explicitly

This change requires changing app-processes.yml to include a list of management comamnds to run

2018-07-18 Upgrade to Python 3

This change installs Python 3.6.8, builds a new virtualenv, and runs CommCare HQ in Python 3.

2019-05-13 Install Pango

This change installs pango and its dependencies for the weasyprint library which has been added as a requirement to commcare-hq for proper pdf printing of unicode fonts

2019-02-26 Fix to restart nginx after every letsencrypt cert auto-renewal

Previously you had to manually restart nginx every time letsencrypt auto-renewed, which was about every two months. We believed we had fixed this with Restart nginx after every letsencrypt cert auto-renewal, but there was an error in our setup at that time that made it not work as intended.

2019-04-05 Update RabbitMQ logging configuration

This change updates the RabbitMQ logging configuration to change the log level from info to warning.

2019-02-27 Remove celery results backend from localsettings

Upgrading to celery 4.x requires removing the dependency on django-celery, which means that its results backend will no longer be available. This removes the django-celery backend as the default from localsettings, so the results backend can be specified by commcare-hq settings instead.

2019-02-26 Split pgbouncer vars from postgresql vars

This change extracts a new role from the existing postgresql role for installing and configuring pgbouncer.

As a result of this change the postgresql.yml environment configuration file needs to be changed to split out the postgresql vars from the pgbouncer vars.

2019-02-27 Only monitor specific RabbitMQ queues

Datadog RabbitMQ monitoring restricts the number of queues it can monitor to 200. To avoid hitting this limit on large scale deployments we limit the queues being monitored to only the primary queues.

2019-02-22 Update supervisor confs to invoke celery directly

Upgrading to celery 4.x requires removing the dependency on django-celery, which means that the celery management command becomes unavailable. This prepares for that by invoking the celery command directly.

2019-02-22 Separate celery datadog http check

This adds a specific http check for the celery check (serverup.txt?only=celery) to datadog. Environments that are not relying on datadog for monitoring can ignore this change.

2019-02-11 Add tag to datadog http checks

This change adds “check_type” tag to the http_check datadog integration. This change applies only to envs using datadog for monitoring.

2019-02-11 Java upgrade for formplayer

Previously, Formplayer was running on Java 7. This change updates us to Java 8 for formplayer.

2019-02-01 Generalize load case from fixture feature

Previously loading a case from a fixture required the fixture to be an attribute. This change allows using non-attributes from the fixture.

2019-01-16 Fix encrypted temp directory permissions

This is a followup to Added encrypted temporary directory in which we introduced an encrypted directory for temp files. In its original implementation, this file was owned by root, and processes were unable to write to it.

This changes the directory to be owned by cchq, allowing our processes to write to the file.

2019-01-02 Restart nginx after every letsencrypt cert auto-renewal

Update 2019-02-26: There was a bug in this fix and it has been superceded by Fix to restart nginx after every letsencrypt cert auto-renewal.

Previously you had to manually restart nginx every time letsencrypt auto-renewed, which was about every two months.

2018-12-15 Blob Metadata Migration - part 2

Form submission attachment metadata is being consolidated in the blob metadata table in SQL. This migration consists of a series of commands that will consolidate the data in your environment.

2018-09-24 Blob Metadata Migration - part 1

Blob metadata needs to be migrated from CouchDB to SQL. This migration consists of a series of commands that will move the data in your environment.

2018-11-26 Reorganize pillows

Pillows read changes from kafka and do various processing such as sending them to elasticsearch, transforming into a UCR table row etc. A doc for same change is read multiple times for each processor, since there are separte pillows for each processor. This is inefficient, so we have combined multiple processors that apply for a given document type (also called KAFKA_TOPIC) such as form/case/user under one pillow. For e.g. A new single case-pillow pillow replaces various old pillows that process case changes such as CaseToElasticsearchPillow, CaseSearchToElasticsearchPillow, ReportCaseToElasticsearchPillow, and kafka-ucr-main etc.

2018-11-20 New Case Importer Celery Queue

Importing cases is often a time-sensitive task, and prolonged backlogs are very visible to users. It will be useful to have a separate queue specifically for case imports, to improve visibility into backups as well as typical runtimes. Additionally, this is a first step towards allocating resources specifically for case imports, should that become necessary.

2018-08-16 Support multiple Kafka brokers

Large scale deployments of CommCare require scaling out Kafka brokers to support the high traffic volume (as well as for high availability). Up until now CommCare has only supported a single broker.

2018-08-16 Add queue for analytics tasks

Tasks for analytics reporting have been separated into a new analytics celery queue.

2018-07-25 Update Supervisor

Ubuntu 14.04 apt-get install supervisor installs supervisor 3.0b. We occasionally have issues that could be related to supervisor, such as processes not stopping correctly. To rule it out as a possible cause, we decided it was better to be on a later version of supervisor, and one that’s not in beta.

2018-07-13 Update supervisor service definitions

There are several CommCare specific processes that are defined in supervisor configuration files. This change decouples the process definitions from code.

2018-06-11 Added encrypted temporary directory

Some of the CommCare processes make use of temporary files to store client data (such as data exports) so in order to keep that data protected we have modified the setup to use an encrypted temporary directory.