Description

The data source is the Pure server (ftp.credit360.com), which Talend will load the file via FTP (more detail from PURE click on this link).  The Talend project EHS_PURE load these data first and keep to GCP project solvay-ind-conso-[env] on the dataset ehs_pure_[env]_mig, which is existing and not part of Operational Dashboard project. 

Source FTP server = "ftp.credit360.com"  / User = "solvay" by using Private key file and it keep in remote engine GCP at this folder \DATA\DEV\EHS\Pure\InOut\pure_sftp_ssh_key (control by context variable l_CNX_EHS_PURE_SFTP_private_key )

Talend Project = EHS_PURE

Talend jobs  = F004_Connect_to_SFTP  + F005_Data_Prep, which are not part of this project.

Load following file  to /DATA/DEV/EHS/Pure/Tmp

GCS folder = ehs_pure_dev_mig

Talend Plan = PL_EHS_PURE_HOURL_New run every hour in PROD

Then, Operation Dashboard project using these data to prj-data-industrial-dash-[dev] project by create views

GCP dataset = solvay-ind-conso-dev.DS_prj_data_industrial_dash

V_core_hd_monthly

V_os_data

V_ps_data

Note: OS =  Occupational safety incidents

          PS = Process Safety

After that project IND_DASHBOARD just create views in dataset prj-data-industrial-dash-dev.DataOcean_solvay_conso → DPL.  There is no Talend jobs.