Clone Tools
  • last updated a few seconds ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
OLMIS-6858 Add cron container to reporting stack
OLMIS-6858 Add cron container to reporting stack
Feel free to notify anyone else who should look at this review.

Feel free to notify anyone else who should look at this review.

There are two main connectors for each service (referencedata and requisition), one source (e.g. source-requisition) and one sink (e.g. sink-requisition). The additional sink connectors are for tab...

There are two main connectors for each service (referencedata and requisition), one source (e.g. source-requisition) and one sink (e.g. sink-requisition). The additional sink connectors are for tables that do not have a single id field defined as the primary key.

This is just to match the schema changes in the db.

This is just to match the schema changes in the db.

Script for starting up the stack locally. Not sure if it works in Linux and older versions of macOS.

Script for starting up the stack locally. Not sure if it works in Linux and older versions of macOS.

New settings to define access to the source db and the sink db (for the connectors).

New settings to define access to the source db and the sink db (for the connectors).

This is just for local dev, removing scalyr and logging to syslog, so the logs can be viewed through docker logs.

This is just for local dev, removing scalyr and logging to syslog, so the logs can be viewed through docker logs.

I don't expect we would start up with different versions for Kafka and Zookeeper, so there is just one Confluent version.

I don't expect we would start up with different versions for Kafka and Zookeeper, so there is just one Confluent version.

This is basically the schema in reference data and requisition, minus the foreign keys. Additionally, there is an additional table to define timeliness for each country, and then the materialized v...

This is basically the schema in reference data and requisition, minus the foreign keys. Additionally, there is an additional table to define timeliness for each country, and then the materialized views at the end.

These are here so that the folders exist when mounting the volume to the container.

These are here so that the folders exist when mounting the volume to the container.

OLMIS-6858 Add heartbeat setting to Postgres connector

So that Kafka is notified and changes are flushed, allowing Postgres to update the WAL's LSN. This will help to avoid storage issues.

OLMIS-6858 Clean up unused Nifi flows

Also, consolidate logging to syslog.

OLMIS-6858 Add cron container to reporting stack

To be able to refresh materialized views with a cronjob. Configured for every 15 minutes.

    • -0
    • +40
    ./cron/periodic/15min/refresh-mv
    • -0
    • +0
    ./cron/periodic/daily/.gitignore
    • -0
    • +0
    ./cron/periodic/hourly/.gitignore
    • -0
    • +0
    ./cron/periodic/monthly/.gitignore
    • -0
    • +0
    ./cron/periodic/weekly/.gitignore
COV-75 - Moving covid-reporting to covid repository

  1. … 48 more files in changeset.
Check Superset health check URL

From login to health, which appears to be what was intended.

Add superset patchup version to env file

So that it is set.

OLMIS-6858 Update sample settings and remove local network

Local network settings should only be on the local docker-compose. Add new settings in the sample.

OLMIS-6858 Improve the database definition of the reporting database

For Superset. Also add some helper scripts for local development.

    • -0
    • +168
    ./docker-compose.local.yml
OLMIS-6858 Get materialized views to work with new Kafka tables

The new tables in the reporting database are being populated by the streaming architecture, now the materialized views should use them.

    • -0
    • +17
    ./config/services/connect/sink-orderables.json
OLMIS-6858 Add requisition tables to data pump

Also, define two replication slots, one for reference data and one for requisition. This isn't technically necessary for the current architecture, but it creates a logical separation for the services.

    • -0
    • +17
    ./config/services/connect/sink-requisition.json
    • -0
    • +27
    ./config/services/connect/source-requisition.json
OLMIS-6858 Enable streaming architecture for reporting stack

This is to start the data pump feature for the reporting stack. The components of the streaming architecture are added (Kafka, Connect, etc.) and the first set of connectors (for reference data) are registered. There is one source connector for all reference data tables and one sink connector for most reference data tables. The additional sink connectors are for tables that are not identified by an id field, but rather by a combination of ids.

    • -0
    • +52
    ./config/services/connect/register.sh
    • -0
    • +17
    ./config/services/connect/sink-referencedata.json
    • -0
    • +17
    ./config/services/connect/sink-requisition-group-members.json
    • -0
    • +17
    ./config/services/connect/sink-supported-programs.json
    • -0
    • +26
    ./config/services/connect/source-referencedata.json
LGTM

LGTM

FE has to fetch data after a restart. I'll add commits asap.

FE has to fetch data after a restart. I'll add commits asap.

OLMIS-6816: Added missing requisition statuses in reporting database
OLMIS-6816: Added missing requisition statuses in reporting database
OLMIS-6816: Added missing requisition statuses in reporting database

    • -2736
    • +3070
    ./nifi-flows/Requisitions_Connector.xml
OLMIS-6816: Decreased level of nifi logger

This reverts commit 841a9f8d

    • -5
    • +5
    ./config/services/nifi/conf/logback.xml
OLMIS-6816: increased level of nifi logger to DEBUG

    • -5
    • +5
    ./config/services/nifi/conf/logback.xml
OLMIS-6816: Increased cryptography version from 2.4.2 to patchup's 2.6.1