...

• ftp.credit360.com    → GCP solvay-ind-conso-[env]
• Loading by plan PL_EHS_PURE_HOURLY : At 08:00 CET, on day 1, 5, 10, 15, 20, 25, and 30 of the month on project WS_EHS_PURE
  • F004_Connect_to_SFTP
  • F005_Data_Prep

Format

• columnar format (file .dsv)

...

Site around 5000 records

GBU around 500 records

Assessment

...

Data is same to

...

the file from "ftp.credit360.com"

Script to check rolling 12 months

Selectsite_code,gbu_code,yearmonth,rii,working_hours,rolling_12_mth_rii,rolling_12_mth_working_hours
from`prj-data-industrial-dash-dev.DM.FACT_trii_site`
orderbysite_code,gbu_code,yearmonth

Each line of rolling 12 months will sum back 12 lines of grouping site, gbu and yearmonth. 

Image Added

Similar to pse. 

...

Loading

1.1 Incremental Load

...

Criticality

High/Medium/Low

Logging

Data Ocean solvay-ind-conso-[env]

There is no log

Product on prj-data-industrial-dash-[env]

selectjob.job_name, job.meta_start_date, job.meta_execution_id, logs.meta_run_id, logs.meta_source_system, logs.meta_step, logs.meta_status, logs.meta_num_lines, logs.meta_error_linesfromSTG.log_tableslogsjoinSTG.run_jobsjobonlogs.meta_run_id = job.meta_run_id
where  logs.meta_run_idin(SELECTmeta_run_idFROMSTG.run_jobsorderbymeta_start_datedesclimit1000)
andmeta_source_systemin('V_ps_data','V_os_data')
andmeta_step = 'ODS to DM'
andmeta_start_date>  DATE_SUB(CURRENT_TIMESTAMP(), INTERVAL5DAY)
orderbyjob.meta_start_datedesc

Image Added