This page presents the data development documentation for Battery Conductivity for all sites.
This page was copied from Battery. Needs to be updated
The standard ETL areas and processes are described on ALB Data Dev Architecture - General. |
*No Labware for Batteries Conductivity
All the schemas are available on google drive for further edition: |
ELN API → <Extraction> → Local JSON File |
| Spreadsheet | Description | Data Status* |
|---|---|---|
| ELN Conductivity Measurement | This is a standard ELN spreadsheet for all sites | |
| ELN Battery Experiment Properties | This is a standard ELN spreadsheet for all sites |
*Data Status: HOT: Data currently being updated at the source. It should be loaded regularly. COLD: No data changes at the source. It should be loaded just once.
The list of the spreadsheets/tables extracted from JSON files coming from ELN can be found in the Data Mapping of the next section.
| Document Name | Link |
|---|---|
| Battery - ELN Data Model | |
| Battery - ELN Template Documentation |
Lab server → Google Share Drive → <Copy> → Local XML/TXT file |
The instrument files are, most of them, manually added to a Google Drive Shared folder
| Site | Description | Status |
|---|---|---|
| Seoul | ||
| Brussels | ||
| Bollate | ||
| Aubervilliers |
*Data Status: HOT: Data currently being updated at the source. It should be loaded regularly. COLD: No data changes at the source. It should be loaded just once.
Local XML/TXT file → <Load> → Cloud Storage/Staging BigQuery |
The spreadsheet below presents all data transformation between the raw files (extracted files) and a BigQuery Staging table.
The data integration phase for batteries follows the standard approach describe on ALB Data Dev Architecture - General.
Staging BigQuery → <Transform> → ODS BigQuery |
The spreadsheet below presents all data transformations from Staging tables to ODS tables. This steps aims to structure the files in the target table format and checking if the column's type (schema) are conformed.
The following data model presents the tables present on ODS dataset and the relation between them:
ODS BigQuery → <Transform> → DW BigQuery → <Expose> → DM_Conductivity BigQuery |
The spreadsheet below presents all data transformations between the tables on ODS and DM_Conductivity. This steps aims to create views for the Data Visualization.
The following data model presents the tables present on DW/DM dataset and the relation between them:
For the DM dataset (DM_Conductivity), we have the views on the top of the tables before. As defined in the convention, there is no need of the abbreviation "conduct". No need either of Talend jobs.
All the jobs are run in sequence under the follow job and project name on Talend Cloud:
| Project | Job/Flow | Associated TMC Plan |
|---|---|---|
| RnI_ACN_Battery | F010_RnI_ACN_Battery_ELN_IDBS_Orch_Flow | PL_RNI_ACN_BATTERY_CONDUCTIVITY_ELN_DAILY |
| RnI_ACN_Battery | F020_RnI_ACN_Battery_ELN_Integration_Orch_Flow | PL_RNI_ACN_BATTERY_CONDUCTIVITY_ELN_DAILY |
| RnI_ACN_Battery | F011_RnI_ACN_Battery_Instruments_Orch_Flow | PL_RNI_ACN_BATTERY_CONDUCTIVITY_INSTRUMENT_DAILY |
| RnI_ACN_Battery | F021_RnI_ACN_Battery_Instr_Integration_Orch_Flow | PL_RNI_ACN_BATTERY_CONDUCTIVITY_INSTRUMENT_DAILY |
For scheduling details check the Operational documentation.
The following jobs should not be orchestrated and only run once during the deployment :
| Project | Job/Flow | Associated TMC Plan |
|---|---|---|
| RnI_ACN_Battery | F010_RnI_ACN_Battery_Create_BQ_Views | PL_RNI_ACN_BATTERY_CONDUCTIVITY_CREATE_VIEWS |
Conductivity (GCP and Tableau documentation) :