DFS Environments

This document provide an overview of the environment and main tools and services.

GCP Project

Environment	URL to Google Cloud Console
Dev	https://console.cloud.google.com/home/dashboard?project=solvay-silica-synthesis-dev
UAT	https://console.cloud.google.com/home/dashboard?project=solvay-silica-synthesis-dev	datasets and buckets suffixed with "_test"
PreProd	https://console.cloud.google.com/home/dashboard?project=solvay-silica-synthesis-prod	datasets and buckets suffixed with "_preprod"
Prod	https://console.cloud.google.com/home/dashboard?project=solvay-silica-synthesis-prod

For Silica, only two kinds of buckets are used for the moment and they are shared between all labs:

[...]-files

These buckets hold the raw data used as base for tables available on the BigQuery tables. Those files must be kept in case of reprocessing.

[...]working-files

This kind of bucket is used by Talend in order to load the data into BigQuery. Those are temporary files and they are overwritten every run.

These are the three Talend projects used for Silica. After migration to Talend Cloud version 8, "-mig" was added to the repo name.

Lab	Talend Project Name	Git
Analytical	RnI_Silica_Analytical-mig	https://gitlab.solvay.com/solvay-it-bda/talend/rni/silica/bda-talend-rni-silica-analytical-mig
Application	RnI_Silica_Application-mig	https://gitlab.solvay.com/solvay-it-bda/talend/rni/silica/bda-talend-rni-silica-application-mig
Synthesis	RnI_Silica_Synthesis-mig	https://gitlab.solvay.com/solvay-it-bda/talend/rni/silica/bda-talend-rni-silica-synthesis-mig

All the scripts are available in a just one Git repo: