This page describes the general process to extract, ingest, prepare, integrate and expose the data on dashboards and reports.

Summary

Macro Diagram

1 - Data Ingestion

Data ingestion is the process of transporting data from one or more sources to a target site for further processing and analysis. For this project data is extracted or copied to Talend server as files using Talend.

ELN

Many spreadsheets are extracted from Electronic Laboratory Notebook (ELN) which is responsable to gather lab documents and result tests. It gives scientists a common view of data across disparate research areas, enabling complete visibility of research information.

ELN data is extracted in JSON format using an available API. 

Related documents: 

ZIFO - API Documentation for extracting E-Workbook Spreadsheet Data

Labware

LabWare LIMS is a laboratory information management system (LIMS) that is "configurable or pre-configured for laboratories of all types and sizes." The software consists of the core LIMS application with access to LabWare's library of LIMS software modules.

Instruments/Cyclers

The instruments files are available in different formats (txt, csv, xml, etc.) usually on Lab server folders or Google drive. The rules to structure the files in columns are specific for every workflow/method. 

2 - Data Preparation

Data preparation is the process of cleaning and transforming raw data prior to processing and analysis. Often involves reformatting data and making corrections to data.

Talend jobs are responsable for cleaning and structuring the data. 

(From Local Files to BigQuery Staging Dataset)

3 - Data Integration and Enrich

Data enrichment refers to the process of appending or otherwise enhancing collected data with relevant context obtained from additional sources. 

(From BigQuery Staging to ODS et DW datasets)

4 - Data Presentation

(Data ready available on DW/DM)

5- Data Visualization

(Tableau dashboards on DW/DM data)

Data Access

The access to the data is controlled using...