This page describes the general process to extract, ingest, prepare, integrate and expose the data on dashboards and reports.

Summary

Macro Diagram

Data Sources

ELN

Labware

Instruments

Data Ingestion

Data ingestion is the process of transporting data from one or more sources to a target site for further processing and analysis. For this project data is extracted or copied using Talend.

(From Source to Local Files)

See ACN document: 

Data Preparation

Data preparation is the process of cleaning and transforming raw data prior to processing and analysis. Often involves reformatting data and making corrections to data.

Talend jobs are responsable for cleaning and structuring the data. 

(From Local Files to BigQuery Staging Dataset)

Data Integration and Enrich

Data enrichment refers to the process of appending or otherwise enhancing collected data with relevant context obtained from additional sources. 

(From BigQuery Staging to ODS et DW datasets)

Data Visualization

(Tableau dashboards on ODS/DW data)

Data Access

The access to the data is controlled using...