Data, Analytics & AI Platform

Page tree


You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

Below are the list of Talend jobs which were developed as part of IRM .

Talend Project :  DATA_OCEAN_DOMAIN_PROCUREMENT

GCP Project : prj-data-dm-procurement-prod

Schedule : All this jobs are scheduled to run weekly (yet the moment its configured to run on Monday's , but can be changed based on the need)

Job Explanation: Available at the bottom of the page.


Source SAP SystemSource TableLoad TypePlan nameTask NameSTG TableODS Table

PF1

EINAFullPL_SPF_F001_F_W_EINA_TO_BQF001_SPF_F001_F_W_EINA_TO_BQSTG_SPF_0000_0000_F001_F_W_EINAODS_SPF_0000_F001_F_W_EINA
EINEFullPL_SPF_F001_F_W_EINE_TO_BQF001_SPF_F001_F_W_EINE_TO_BQSTG_SPF_0000_0000_F001_F_W_EINEODS_SPF_0000_F001_F_W_EINE
EKKOIncrementalPL_SPF_F001_I_W_EKKO_TO_BQF001_SPF_F001_I_W_EKKO_TO_BQSTG_SPF_0000_0000_F001_I_W_EKKOODS_SPF_0000_F001_I_W_EKKO
ISEGIncrementalPL_SPF_F001_I_W_ISEG_TO_BQF001_SPF_F001_I_W_ISEG_TO_BQSTG_SPF_0000_0000_F001_I_W_ISEGODS_SPF_0000_F001_I_W_ISEG
LFB1FullPL_SPF_F001_F_W_LFB1_TO_BQF001_SPF_F001_F_W_LFB1_TO_BQSTG_SPF_0000_0000_F001_F_W_LFB1ODS_SPF_0000_F001_F_W_LFB1
LFM1FullPL_SPF_F001_F_W_LFM1_TO_BQF001_SPF_F001_F_W_LFM1_TO_BQSTG_SPF_0000_0000_F001_F_W_LFM1ODS_SPF_0000_F001_F_W_LFM1
LFM2FullPL_SPF_F001_F_W_LFM2_TO_BQF001_SPF_F001_F_W_LFM2_TO_BQSTG_SPF_0000_0000_F001_F_W_LFM2ODS_SPF_0000_F001_F_W_LFM2
MARAFullPL_SPF_F001_F_W_MARA_TO_BQF001_SPF_F001_F_W_MARA_TO_BQSTG_SPF_0000_0000_F001_F_W_MARAODS_SPF_0000_F001_F_W_MARA
MARCFullPL_SPF_F001_F_W_MARC_TO_BQF001_SPF_F001_F_W_MARC_TO_BQSTG_SPF_0000_0000_F001_F_W_MARCODS_SPF_0000_F001_F_W_MARC
MARDFullPL_SPF_F001_F_W_MARD_TO_BQF001_SPF_F001_F_W_MARD_TO_BQSTG_SPF_0000_0000_F001_F_W_MARDODS_SPF_0000_F001_F_W_MARD
MASTFullPL_SPF_F001_F_W_MAST_TO_BQF001_SPF_F001_F_W_MAST_TO_BQSTG_SPF_0000_0000_F001_F_W_MASTODS_SPF_0000_F001_F_W_MAST
MCHBIncrementalPL_SPF_F001_I_W_MCHB_TO_BQF001_SPF_F001_I_W_MCHB_TO_BQSTG_SPF_0000_0000_F001_I_W_MCHBODS_SPF_0000_F001_I_W_MCHB
MKALFullPL_SPF_F001_F_W_MKAL_TO_BQF001_SPF_F001_F_W_MKAL_TO_BQSTG_SPF_0000_0000_F001_F_W_MKALODS_SPF_0000_F001_F_W_MKAL
MKPFIncrementalPL_SPF_F001_I_W_MKPF_TO_BQF001_SPF_F001_I_W_MKPF_TO_BQSTG_SPF_0000_0000_F001_I_W_MKPFODS_SPF_0000_F001_I_W_MKPF
MSEGIncrementalPL_SPF_F001_I_W_MSEG_TO_BQF001_SPF_F001_I_W_MSEG_TO_BQSTG_SPF_0000_0000_F001_I_W_MSEGODS_SPF_0000_F001_I_W_MSEG
STPOFullPL_SPF_F001_F_W_STPO_TO_BQF001_SPF_F001_F_W_STPO_TO_BQSTG_SPF_0000_0000_F001_F_W_STPOODS_SPF_0000_F001_F_W_STPO
T024FullPL_SPF_F001_F_W_T024_TO_BQF001_SPF_F001_F_W_T024_TO_BQSTG_SPF_0000_0000_F001_F_W_T024ODS_SPF_0000_F001_F_W_T024
ASMDFullPL_SPF_F001_F_W_ASMD_TO_BQF001_SPF_F001_F_W_ASMD_TO_BQSTG_SPF_0000_0000_F001_F_W_ASMDODS_SPF_0000_F001_F_W_ASMD
EBANIncrementalPL_SPF_F001_I_W_EBAN_TO_BQF001_SPF_F001_I_W_EBAN_TO_BQSTG_SPF_0000_0000_F001_I_W_EBANODS_SPF_0000_F001_I_W_EBAN
EKPOIncrementalPL_SPF_F001_I_W_EKPO_TO_BQF001_SPF_F001_I_W_EKPO_TO_BQSTG_SPF_0000_0000_F001_I_W_EKPOODS_SPF_0000_F001_I_W_EKPO
ESLHFullPL_SPF_F001_F_W_ESLH_TO_BQF001_SPF_F001_F_W_ESLH_TO_BQSTG_SPF_0000_0000_F001_F_W_ESLHODS_SPF_0000_F001_F_W_ESLH
ESLL *FullPL_SPF_F001_F_W_ESLL_TO_BQF001_SPF_F001_F_W_ESLL_TO_BQSTG_SPF_0000_0000_F001_F_W_ESLLODS_SPF_0000_F001_F_W_ESLL
ESSRIncrementalPL_SPF_F001_I_W_ESSR_TO_BQF001_SPF_F001_I_W_ESSR_TO_BQSTG_SPF_0000_0000_F001_I_W_ESSRODS_SPF_0000_F001_I_W_ESSR
MBEWFullPL_SPF_F001_F_W_MBEW_TO_BQF001_SPF_F001_F_W_MBEW_TO_BQSTG_SPF_0000_0000_F001_F_W_MBEWODS_SPF_0000_F001_F_W_MBEW
T024WFullPL_SPF_F001_F_W_T024W_TO_BQF001_SPF_F001_F_W_T024W_TO_BQSTG_SPF_0000_0000_F001_F_W_T024WODS_SPF_0000_F001_F_W_T024W
MAKTFullPL_SPF_F001_F_W_MAKT_TO_BQF001_SPF_F001_F_W_MAKT_TO_BQSTG_SPF_0000_0000_F001_F_W_MAKTODS_SPF_0000_F001_F_W_MAKT
T156TFullPL_SPF_F001_F_W_T156T_TO_BQF001_SPF_F001_F_W_T156T_TO_BQSTG_SPF_0000_0000_F001_F_W_T156TODS_SPF_0000_F001_F_W_T156T
T001WFullPL_SPF_F001_F_W_T001W_TO_BQF001_SPF_F001_F_W_T001W_TO_BQSTG_SPF_0000_0000_F001_F_W_T001WODS_SPF_0000_F001_F_W_T001W
T001LFullPL_SPF_F001_F_W_T001L_TO_BQF001_SPF_F001_F_W_T001L_TO_BQSTG_SPF_0000_0000_F001_F_W_T001LODS_SPF_0000_F001_F_W_T001L
T001FullPL_SPF_F001_F_W_T001_TO_BQF001_SPF_F001_F_W_T001_TO_BQSTG_SPF_0000_0000_F001_F_W_T001ODS_SPF_0000_F001_F_W_T001
T064TFullPL_SPF_F001_F_W_T064T_TO_BQF001_SPF_F001_F_W_T064T_TO_BQSTG_SPF_0000_0000_F001_F_W_T064TODS_SPF_0000_F001_F_W_T064T


Source SAP SystemSource TableLoad TypePlan nameTask NameSTG TableODS Table

WP1

EINAFullPL_SPW_F001_F_W_EINA_TO_BQF001_SPW_F001_F_W_EINA_TO_BQSTG_SPW_0000_0000_F001_F_W_EINAODS_SPW_0000_F001_F_W_EINA
EINEFullPL_SPW_F001_F_W_EINE_TO_BQF001_SPW_F001_F_W_EINE_TO_BQSTG_SPW_0000_0000_F001_F_W_EINEODS_SPW_0000_F001_F_W_EINE
EKKOIncrementalPL_SPW_F001_I_W_EKKO_TO_BQF001_SPW_F001_I_W_EKKO_TO_BQSTG_SPW_0000_0000_F001_I_W_EKKOODS_SPW_0000_F001_I_W_EKKO
ISEGIncrementalPL_SPW_F001_I_W_ISEG_TO_BQF001_SPW_F001_I_W_ISEG_TO_BQSTG_SPW_0000_0000_F001_I_W_ISEGODS_SPW_0000_F001_I_W_ISEG
LFB1FullPL_SPW_F001_F_W_LFB1_TO_BQF001_SPW_F001_F_W_LFB1_TO_BQSTG_SPW_0000_0000_F001_F_W_LFB1ODS_SPW_0000_F001_F_W_LFB1
LFM1FullPL_SPW_F001_F_W_LFM1_TO_BQF001_SPW_F001_F_W_LFM1_TO_BQSTG_SPW_0000_0000_F001_F_W_LFM1ODS_SPW_0000_F001_F_W_LFM1
LFM2FullPL_SPW_F001_F_W_LFM2_TO_BQF001_SPW_F001_F_W_LFM2_TO_BQSTG_SPW_0000_0000_F001_F_W_LFM2ODS_SPW_0000_F001_F_W_LFM2
MARAFullPL_SPW_F001_F_W_MARA_TO_BQF001_SPW_F001_F_W_MARA_TO_BQSTG_SPW_0000_0000_F001_F_W_MARAODS_SPW_0000_F001_F_W_MARA
MARCFullPL_SPW_F001_F_W_MARC_TO_BQF001_SPW_F001_F_W_MARC_TO_BQSTG_SPW_0000_0000_F001_F_W_MARCODS_SPW_0000_F001_F_W_MARC
MARDFullPL_SPW_F001_F_W_MARD_TO_BQF001_SPW_F001_F_W_MARD_TO_BQSTG_SPW_0000_0000_F001_F_W_MARDODS_SPW_0000_F001_F_W_MARD
MASTFullPL_SPW_F001_F_W_MAST_TO_BQF001_SPW_F001_F_W_MAST_TO_BQSTG_SPW_0000_0000_F001_F_W_MASTODS_SPW_0000_F001_F_W_MAST
MCHBIncrementalPL_SPW_F001_I_W_MCHB_TO_BQF001_SPW_F001_I_W_MCHB_TO_BQSTG_SPW_0000_0000_F001_I_W_MCHBODS_SPW_0000_F001_I_W_MCHB
MKALFullPL_SPW_F001_F_W_MKAL_TO_BQF001_SPW_F001_F_W_MKAL_TO_BQSTG_SPW_0000_0000_F001_F_W_MKALODS_SPW_0000_F001_F_W_MKAL
MKPFIncrementalPL_SPW_F001_I_W_MKPF_TO_BQF001_SPW_F001_I_W_MKPF_TO_BQSTG_SPW_0000_0000_F001_I_W_MKPFODS_SPW_0000_F001_I_W_MKPF
MSEGIncrementalPL_SPW_F001_I_W_MSEG_TO_BQF001_SPW_F001_I_W_MSEG_TO_BQSTG_SPW_0000_0000_F001_I_W_MSEGODS_SPW_0000_F001_I_W_MSEG
STPOFullPL_SPW_F001_F_W_STPO_TO_BQF001_SPW_F001_F_W_STPO_TO_BQSTG_SPW_0000_0000_F001_F_W_STPOODS_SPW_0000_F001_F_W_STPO
T024FullPL_SPW_F001_F_W_T024_TO_BQF001_SPW_F001_F_W_T024_TO_BQSTG_SPW_0000_0000_F001_F_W_T024ODS_SPW_0000_F001_F_W_T024
ASMDFullPL_SPW_F001_F_W_ASMD_TO_BQF001_SPW_F001_F_W_ASMD_TO_BQSTG_SPW_0000_0000_F001_F_W_ASMDODS_SPW_0000_F001_F_W_ASMD
EBANIncrementalPL_SPW_F001_I_W_EBAN_TO_BQF001_SPW_F001_I_W_EBAN_TO_BQSTG_SPW_0000_0000_F001_I_W_EBANODS_SPW_0000_F001_I_W_EBAN
EKPOIncrementalPL_SPW_F001_I_W_EKPO_TO_BQF001_SPW_F001_I_W_EKPO_TO_BQSTG_SPW_0000_0000_F001_I_W_EKPOODS_SPW_0000_F001_I_W_EKPO
ESLHFullPL_SPW_F001_F_W_ESLH_TO_BQF001_SPW_F001_F_W_ESLH_TO_BQSTG_SPW_0000_0000_F001_F_W_ESLHODS_SPW_0000_F001_F_W_ESLH
ESLLFullPL_SPW_F001_F_W_ESLL_TO_BQF001_SPW_F001_F_W_ESLL_TO_BQSTG_SPW_0000_0000_F001_F_W_ESLLODS_SPW_0000_F001_F_W_ESLL
ESSRIncrementalPL_SPW_F001_I_W_ESSR_TO_BQF001_SPW_F001_I_W_ESSR_TO_BQSTG_SPW_0000_0000_F001_I_W_ESSRODS_SPW_0000_F001_I_W_ESSR
MBEWFullPL_SPW_F001_F_W_MBEW_TO_BQF001_SPW_F001_F_W_MBEW_TO_BQSTG_SPW_0000_0000_F001_F_W_MBEWODS_SPW_0000_F001_F_W_MBEW
T024WFullPL_SPW_F001_F_W_T024W_TO_BQF001_SPW_F001_F_W_T024W_TO_BQSTG_SPW_0000_0000_F001_F_W_T024WODS_SPW_0000_F001_F_W_T024W
MAKTFullPL_SPW_F001_F_W_MAKT_TO_BQF001_SPW_F001_F_W_MAKT_TO_BQSTG_SPW_0000_0000_F001_F_W_MAKTODS_SPW_0000_F001_F_W_MAKT
T156TFullPL_SPW_F001_F_W_T156T_TO_BQF001_SPW_F001_F_W_T156T_TO_BQSTG_SPW_0000_0000_F001_F_W_T156TODS_SPW_0000_F001_F_W_T156T
T001WFullPL_SPW_F001_F_W_T001W_TO_BQF001_SPW_F001_F_W_T001W_TO_BQSTG_SPW_0000_0000_F001_F_W_T001WODS_SPW_0000_F001_F_W_T001W
T001LFullPL_SPW_F001_F_W_T001L_TO_BQF001_SPW_F001_F_W_T001L_TO_BQSTG_SPW_0000_0000_F001_F_W_T001LODS_SPW_0000_F001_F_W_T001L
T001FullPL_SPW_F001_F_W_T001_TO_BQF001_SPW_F001_F_W_T001_TO_BQSTG_SPW_0000_0000_F001_F_W_T001ODS_SPW_0000_F001_F_W_T001
T064TFullPL_SPW_F001_F_W_T064T_TO_BQF001_SPW_F001_F_W_T064T_TO_BQSTG_SPW_0000_0000_F001_F_W_T064TODS_SPW_0000_F001_F_W_T064T


We are using the Data Ocean standard jobs to extract and load the data into the STG/ODS tables. As we are dealing with Full and incremental type of loading so there is a little variance in the jobs, which is explained below:


Full Load

For the explanation, I am using F001_SPF_F001_F_W_ESLL_TO_BQ which is a full load job.  Job parameters like STG table, ODS table , Bucket details , filters etc are maintained in the RDS parameters and its not hardcoded in the job, if required it can be changed any time in the RDS itself.

Parameters Snapshot

  • All parameters are self explanatory except the additional_filter which is used to define the type of the load i.e. blank means its a full refresh and "incremental" mean its a incremental load type.  See the below example for PF1_MSEG which is a incremental load type, so we used "incremental" in the additional_filter parameter.


Job Details

Below is the snapshot of the main Flow job, where we are using all the Data Ocean standard jobs , except "SAP to GCS" extraction based on each source.

J020_SPF_Table_ESLL_to_GCS  Job which is used to extract the data from the SAP table and load into the Google Cloud Storage, as I said all the parameters are configured in the RDS and no hardcoding of any values in the Talend job.


Incremental Load

For the explanation, I am using F001_SPF_F001_F_W_ESLL_TO_BQ which is a full load job.  Job parameters like STG table, ODS table , Bucket details , filters etc are maintained in the RDS parameters and its not hardcoded in the job, if required it can be changed any time in the RDS itself.

Parameters Snapshot

  • All parameters are self explanatory except the additional_filter which is used to define the type of the load i.e. blank means its a full refresh and "incremental" mean its a incremental load type.  See the below example for PF1_MSEG which is a incremental load type, so we used "incremental" in the additional_filter parameter.


Job Details

Below is the snapshot of the main Flow job, where we are using all the Data Ocean standard jobs , except "SAP to GCS" extraction based on each source.

J020_SPF_Table_ESLL_to_GCS  Job which is used to extract the data from the SAP table and load into the Google Cloud Storage, as I said all the parameters are configured in the RDS and no hardcoding of any values in the Talend job.




Powered by Atlassian Confluence and Brikit Theme Press