Clone
 

chongsun ahn <cahn922@yahoo.com> in openlmis-ref-distro

OLMIS-6858 Remove unnecessary files

Remove jolt spec files and Nifi XML templates, as they are no longer used in Nifi.

    • -4626
    • +0
    /reporting/nifi-flows/OpenLMIS_v3_reference_data.xml
OLMIS-6858 Add heartbeat setting to Postgres connector

So that Kafka is notified and changes are flushed, allowing Postgres to update the WAL's LSN. This will help to avoid storage issues.

OLMIS-6858 Clean up unused Nifi flows

Also, consolidate logging to syslog.

    • -1
    • +1
    /reporting/cron/periodic/15min/refresh-mv
OLMIS-6858 Add cron container to reporting stack

To be able to refresh materialized views with a cronjob. Configured for every 15 minutes.

    • -0
    • +5
    /reporting/cron/Dockerfile
    • -0
    • +40
    /reporting/cron/periodic/15min/refresh-mv
    • -0
    • +0
    /reporting/cron/periodic/daily/.gitignore
    • -0
    • +0
    /reporting/cron/periodic/hourly/.gitignore
    • -0
    • +0
    /reporting/cron/periodic/monthly/.gitignore
    • -0
    • +0
    /reporting/cron/periodic/weekly/.gitignore
    • -0
    • +16
    /reporting/docker-compose.local.yml
Merge branch 'OLMIS-6858-data-pump-reporting'

Check Superset health check URL

From login to health, which appears to be what was intended.

Add superset patchup version to env file

So that it is set.

OLMIS-6858 Update sample settings and remove local network

Local network settings should only be on the local docker-compose. Add new settings in the sample.

OLMIS-6858 Improve the database definition of the reporting database

For Superset. Also add some helper scripts for local development.

    • -0
    • +168
    /reporting/docker-compose.local.yml
    • -0
    • +57
    /reporting/start-local.sh
OLMIS-6858 Get materialized views to work with new Kafka tables

The new tables in the reporting database are being populated by the streaming architecture, now the materialized views should use them.

    • -0
    • +17
    /reporting/config/services/connect/sink-orderables.json
OLMIS-6858 Add requisition tables to data pump

Also, define two replication slots, one for reference data and one for requisition. This isn't technically necessary for the current architecture, but it creates a logical separation for the services.

OLMIS-6858 Enable streaming architecture for reporting stack

This is to start the data pump feature for the reporting stack. The components of the streaming architecture are added (Kafka, Connect, etc.) and the first set of connectors (for reference data) are registered. There is one source connector for all reference data tables and one sink connector for most reference data tables. The additional sink connectors are for tables that are not identified by an id field, but rather by a combination of ids.

    • -0
    • +52
    /reporting/config/services/connect/register.sh
    • -0
    • +17
    /reporting/config/services/connect/sink-supported-programs.json
    • -0
    • +26
    /reporting/config/services/connect/source-referencedata.json
Update some services to next minor versions

Update docs on service health to new endpoint

Change ERD generation to Debezium image

OLMIS-6761 Set superset patchup version to latest version

To fix OAuth bug.

OLMIS-6771 Bump diagnostics to next minor version

Upgrade OpenLMIS Debezium Connect image

To be based on Debezium Connect 1.0.

    • -2
    • +2
    /reporting/debezium-connect/Dockerfile
Create toolbelt Docker image

For streaming pipeline.

    • -0
    • +5
    /utils/toolbelt/Dockerfile
    • -0
    • +3
    /utils/toolbelt/README.md
OLMIS-6764 Bump notification to next minor version

OLMIS-6761 Bump auth to next minor version

OLMIS-6760 Bump report version to next minor version

OLMIS-6514 Update OpenLMIS Debezium Connect docker image version

To go to latest available for new features (like better delete support).

    • -3
    • +3
    /reporting/debezium-connect/Dockerfile
Update Casper documentation

The pipeline has been streamlined.

Add stock adjustments to transform pipeline

* Add stock adjustments to connectors

* Add XML template

* Add properties file to autoload from Nifi registry

* Add schema mapping file

* Add adjustment reason mapping file

    • -0
    • +4573
    /reporting/TransformReqLineItemLossAdjV2ToV3.xml
    • -1
    • +1
    /reporting/casper-register-v2-source.json
    • -1
    • +1
    /reporting/casper-register-v3-sink.json
Minor changes to transform XML templates

    • -1
    • +1
    /reporting/TransformReqLineItemsV2ToV3.xml
    • -1
    • +1
    /reporting/TransformReqStatusChangesV2ToV3.xml
Include requisition status mapping

Release no order should be mapped to release without order.

Add requisition comments/status messages to transform pipeline

* Add comments/status message tables to connectors

* Add XML template

* Add properties file to autoload from Nifi registry

* Add schema mapping file

    • -0
    • +6054
    /reporting/TransformReqStatusMessagesV2ToV3.xml
    • -1
    • +1
    /reporting/casper-register-v2-source.json
    • -1
    • +1
    /reporting/casper-register-v3-sink.json
Add requisition status changes to transform pipeline

* Add status changes tables to connectors

* Add XML template

* Add properties file to autoload from Nifi registry

* Add schema mapping file

* Technically RELEASED_NO_ORDER is not in the schema, but in the payload, but it is easier to include in the schema mapping file.

    • -0
    • +4311
    /reporting/TransformReqStatusChangesV2ToV3.xml
    • -1
    • +1
    /reporting/casper-register-v2-source.json
    • -1
    • +1
    /reporting/casper-register-v3-sink.json
Add step in transform requisitions for regen perms

Regenerate requisition permission strings when new requisition is inserted.