Talend Data Flow Documentation
Project: Specialty Polymers
────────────────────────────────────────
Source: Labwer (Shared Folders on NAT) → Target: GCP BigQuery Tables
Data Engineering Team
This document provides the technical documentation for the Talend data pipeline implemented under the Specialty Polymers project. It covers the end-to-end data flow from the source system (Labwer shared folders deployed on a NAT environment) through to the target destination GCP BigQuery tables. It also describes the compression strategy applied by the Product Owner to manage large data volumes efficiently.
The pipeline follows a standard Extract → Transform → Load (ETL) pattern, enriched with a compression/decompression layer to handle the large data volumes characteristic of this project:
Source :
| Category | Item / Source | Details / Target |
| Oracle Sources | labw-p-oracle-01.syensqo.com | |
| Oracle Sources | labw-q-oracle-01.syensqo.com | |
| Bollate File Paths | Source Directory | \\ITBOLVRS06T\Lab Booster |
| Bollate File Paths | Example File Note that .txt files may contain complex and irregular structures, which can make parsing challenging. | \\10.53.6.10\labo\W-524600\TGA\DA CANC\23-11194-6715351-tga-residuo da acque 965pi plx485- aria - sciarrillo.txt |
| Alpharetta File Paths | Test Files Directory | \\USALPACDv02\Test Files\LabBooster |
Talend Extraction & Tmp shared folder:
| Category | Item / Source | Details / Target |
| Talend / Jobs | Orchestration Job | F730_Thermal_Data_Compression_Orch |
| Talend / Jobs | Sub-Job 1 | J125_instrument_Raw_Data_Compressed_To_Bigquery |
| Talend / Jobs | Sub-Job 2 | J125_Thermal_Raw_Data_Compressed_To_Bigquery |
| Talend / Jobs | SQL Queries Path | V:\PROD\RnI\ACN_Materials\SQLQueries |
| Local Storage | Temp Compression Folder | Z:\(ENV)\RnI\ACN_Materials\tmp\Working\data_compression |
GCP (Bigquery & GCS)
| Category | Item / Source | Details / Target |
| Cloud Console | GCP Storage Link | https://console.cloud.google.com/storage/browser/cs-ew1-labboostermaterials-prod-accepted-files/instruments |
| GCP Targets | Staging (compressed) - Delta | gcp-sqo-labbooster-materials-d.Staging.compressed_thermal_raw_data_delta |
| GCP Targets | Staging (compressed) - Conso | gcp-sqo-labbooster-materials-d.Staging.compressed_thermal_raw_data_conso |
| GCP Targets | ODS (compressed) - Delta | gcp-sqo-labbooster-materials-d.ODS.compressed_thermal_raw_data_delta |
| GCP Targets | ODS (compressed) - Conso | gcp-sqo-labbooster-materials-d.ODS.compressed_thermal_raw_data_conso |
Reporting:
| Category | Item / Source | Details / Target |
| Reporting | Tableau Link | https://eu-west-1a.online.tableau.com/#/site/syensqo/views/MaterialsThermalv1_4DSC-TGA/MaterialThermalDSCdev |