Technical documentation for the data coming from reaction tests

Summary

Sum-up

Data Sources

• ELN
• Raw Data

Data Collection

The talend jobs J010_Download_Synthesis_LabServers and J011_Download_Synthesis_gDrive_Reaction extract the raw data files listed on the ELN table synthesis_raw_data_link for which the field “synthesis_equipment_name” is the scale name, i.e. “FR-80L”. For information of how these job works, check the following page : 

Talend - Jobs - Synthesis - Download (needs to be created)

Exemples

Lab servers source

• \\FRPH2-labpc-backup\labo\W-522649\DATAS DATALAKE
• \\FRPH2-LABPC-BACKUP\LABO\W-509931

Python files

• download_filtration170L.py
• download_synthesis25L.py

Output folders

• D:\DATA\[ENV]\RnI\Silica\tmp\Synthesis25L
• D:\DATA\[ENV]\RnI\Silica\tmp\Synthesis170L

Data Preparation

Parse

The parsing python scripts extracts from the raw data files the needed columns.

Columns List

For each sample, the script extracts the many fields from the raw data files and outputs a .csv file. For the mapping details, please refers to the sheet "Parse Mapping" on the Reaction Mapping spreadsheet (link to the spreadsheet on the Sum-up section).

The following columns extracted from the raw data file:

• unique_id
• study_id
• sample_id
• operator
• reactor
• date
• time (in minutes)
• ph
• temperature
• acid_mass_one
• silicate_mass
• additive_mass
• acid_mass_two
• variable_product_mass (empty for 170L scale)
• percent_acid_one
• percent_silicate
• percent_additive
• percent_acid_two
• percent_pump_pH_control
• percent_variable_product (empty for 170L scale)
• turbidity

Compute

The compute python script uses as input the parsed .csv files previously created  and the tables synthesis_eln_data and operating_procedure. It computes the new columns and values from raw data and regenerates new files. 

If the output files already exist the script will NOT replace them.

In the beginning of the script, for each product, we extract the following values for each product listed in the table synthesis_eln_data. Each of the following values will be used in later computations as constants:

Molar masses for the following elements are also defined and used in later computations (mm_ stands for molar mass):

• mm_na2o = 61.98
• mm_h2so4 = 98.079
• mm_sio2 = 60.084
• mm_na2so4 = 142
• mm_hcl = 36.46
• mm_hno3 = 63.02
• mm_hplus = 1.01

Next, we define the activity on each pump as follows:

• We first define “by default” activity on each pump:
• acid_one pump (concentrated acid) → Sulfuric Acid Concentrate
• silicate pump → Silicate
• additive pump → Aluminate
• acid_two pump (diluted acid) → Sulfuric Acid
• Next, from the table operating_procedure, we extract the changes in the activity for each pump. The next table lists the products that can be present on each pump:

For each sample, the compute scripts create three different tables:

ReactionDetails

The first table is composed of the columns previously extracted from the raw data files and the new columns calculated during the execution.

Dataset : raw_data_synthesis

For the columns details, please refers to the sheets "Details Mappings" on the Reaction Mapping spreadsheet (link to the spreadsheet on the Sum-up section).

ReactionSummary

The second table is composed of the new values computed from raw data. This is a atomic table and it aggregates the values by unique_id, study_id and sample_id which represents one line per data raw file.

Dataset : raw_data_synthesis

For the columns details, please refers to the sheets "Summary Mapping" on the Reaction Mapping spreadsheet (link to the spreadsheet on the Sum-up section). 

ReactionMaterialBalance

To be documented

Presentation

The raw data (already parsed) and the computed columns are created as tables on BigQuery. A Talend job is responsable to push all this data to a dataset called raw_data_synthesis.