Versions Compared

Old Version 2

changes.mady.by.user BENSMIDA-ext, Sami

Saved on

compared with

New Version Current

changes.mady.by.user BENSMIDA-ext, Sami

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The pipeline follows a standard Extract → Transform → Load (ETL) pattern, enriched with a compression/decompression layer to handle the large data volumes characteristic of this project:


Image Modified


Source :

CategoryItem / SourceDetails / Target
Oracle Sourceslabw-p-oracle-01.syensqo.com
Oracle Sourceslabw-q-oracle-01.syensqo.com
Bollate File PathsSource Directory\\ITBOLVRS06T\Lab Booster
Bollate File Paths

Example File

Note that .txt files may contain complex and irregular structures, which can make parsing challenging.

\\10.53.6.10\labo\W-524600\TGA\DA CANC\23-11194-6715351-tga-residuo da acque 965pi plx485- aria - sciarrillo.txt
Alpharetta File PathsTest Files Directory\\USALPACDv02\Test Files\LabBooster


Talend Extraction & Tmp shared folder:

CategoryItem / SourceDetails / Target
Talend / JobsOrchestration JobF730_Thermal_Data_Compression_Orch
Talend / JobsSub-Job 1J125_instrument_Raw_Data_Compressed_To_Bigquery
Talend / JobsSub-Job 2J125_Thermal_Raw_Data_Compressed_To_Bigquery
Talend / JobsSQL Queries PathV:\PROD\RnI\ACN_Materials\SQLQueries
Local StorageTemp Compression FolderZ:\(ENV)\RnI\ACN_Materials\tmp\Working\data_compression


GCP (Bigquery & GCS)

CategoryItem / SourceDetails / TargetCloud Console
GCP Storage LinkTargetsStaging (Raw) - Deltagcp-sqo-labbooster-materials-d.Staging.compressed_thermal_raw_data_deltahttps://console.cloud.google.com/storage/browser/cs-ew1-labboostermaterials-prod-accepted-files/instruments
GCP TargetsStaging (compressedRaw) - DeltaConsogcp-sqo-labbooster-materials-d.Staging.compressed_thermal_raw_data_conso
GCP TargetsODS (compressed) - Delta`gcp-sqo-labbooster-materials-p.ODS.compressed_raw_data_deltaconso` 
GCP TargetsStaging ODS (compressed) - Consogcp`gcp-sqo-labbooster-materials-dp.StagingODS.compressed_thermal_raw_data_consodelta` 
GCP TargetsODS (compressed) - Deltagcp`gcp-sqo-labbooster-materials-dp.ODS.compressed_thermal_raw_data_deltaconso` 
GCP TargetsODS (compressed) - Consogcp`gcp-sqo-labbooster-materials-dp.ODS.compressed_thermal_raw_data_delta` 
GCP TargetsODS (Compressed + Filtred data)`gcp-sqo-labbooster-materials-p.ODS.summary_results_conso` 
GCP TargetsODS Raw delta Table `gcp-sqo-labbooster-materials-p.ODS.raw_data_delta` 
GCP TargetsDM- Vue - Used as source for Tableau Software`gcp-sqo-labbooster-materials-p.DM.summary_results`
GCP TargetsEMAIL - ALERT TABLE`gcp-sqo-labbooster-materials-p.DM.ALERT_ERRORS_INSTRUMENTS_FILES` _conso

Reporting:

CategoryItem / SourceDetails / Target
ReportingTableau Linkhttps://eu-west-1a.online.tableau.com/#/site/syensqo/views/MaterialsThermalv1_4DSC-TGA/MaterialThermalDSCdev

3. Alerts:

A dedicated Talend job is responsible for validating input source files before processing.

If a file is detected as corrupted or fails validation checks, the job automatically triggers an alert notification. This alert is sent to the relevant stakeholders to ensure prompt awareness and intervention.

Notifications are delivered via the SMTP server, enabling email-based alerts to communicate issues in near real time.

4.  Contacts & responsibilities: