Description
Tools: Talend
Detail job
- J090_Jira_search_to_GCS
Flow job
- F090_Jira_search_to_ODS
- Setup meta_run_id and filename of the output file
- Get the last load from table STG.incremetnal_load, control by the variable l_VAR_eBatch_PF1_QAPP_INC_LOAD and configuration the logic of the incremental load in tJava to use the date from incremental_load to the field of create or change date in the SAP
- Call the detail job and pass parameters such as user/password, query from point number 2 to do the incremental load and save the file to GCS
- Call the standard job to upload the files from GCS to ODS
- If the loading is OK and parameter l_VAR_heliux_[table_name]_reload = incremental, update the time on the table incremental_load. If the value is not incremental, it is the reloading
- If everything is OK, update the log.
Access rights
Source
Format
- JSON
Destination
Location
The data will keep in
- Bucket = cs-ew1-prj-data-dm-dt-[dev]-staging/xxx
- DataOean GCP = prj-data-dm-dt-[env]
- STG Table name = prj-data-dm-dt-[env].STG.STG_JIR_0000_0000_F001_I_H_JiraSearch
- ODS Table name = prj-data-dm-dt-[Env].ODS.ODS_JIR_0000_F001_I_H_JiraSearch
- DPL View name = To be updated
To save the data into GCP Industrial Data Ocean, service account is required
Format
- columnar format
Sizing
Assessment
How to validate that the generated output is valid:
Loading
1.1 Incremental Load
1.2 Full load
1.3. Reloading data
1.4 Plan to schedule
1.5 Timing
The average time expected for loading:
Criticality
High/Medium/Low