View Source

This document provide an overview of the environment and main tools and services.

GCP Project

Environment	URL to Google Cloud Console
Dev	https://console.cloud.google.com/home/dashboard?project=solvay-silica-synthesis-dev
UAT	https://console.cloud.google.com/home/dashboard?project=solvay-silica-synthesis-dev	datasets and buckets suffixed with "_test"
PreProd	https://console.cloud.google.com/home/dashboard?project=solvay-silica-synthesis-prod	datasets and buckets suffixed with "_preprod"
Prod	https://console.cloud.google.com/home/dashboard?project=solvay-silica-synthesis-prod

Data Solutions - BD&A Team > DFS Environments > image2022-10-13_10-5-7.png

For Silica, only two kinds of buckets are used for the moment and they are shared between all labs:

[...]-files

These buckets hold the raw data used as base for tables available on the BigQuery tables. Those files must be kept in case of reprocessing.

[...]working-files

This kind of bucket is used by Talend in order to load the data into BigQuery. Those are temporary files and they are overwritten every run.

Data Solutions - BD&A Team > DFS Environments > image2022-9-22_14-38-58.png

These are the three Talend projects used for Silica. After migration to Talend Cloud version 8, "-mig" was added to the repo name.

Lab	Talend Project Name	Git
Analytical	RnI_Silica_Analytical-mig	https://gitlab.solvay.com/solvay-it-bda/talend/rni/silica/bda-talend-rni-silica-analytical-mig
Application	RnI_Silica_Application-mig	https://gitlab.solvay.com/solvay-it-bda/talend/rni/silica/bda-talend-rni-silica-application-mig
Synthesis	RnI_Silica_Synthesis-mig	https://gitlab.solvay.com/solvay-it-bda/talend/rni/silica/bda-talend-rni-silica-synthesis-mig

All the scripts are available in a just one Git repo:

Data Solutions - BD&A Team > DFS Environments > image2022-10-17_17-25-28.png