Description

Tools: Talend

Detail job

• J090_Jira_search_to_GCS

Flow job

• F090_Jira_search_to_ODS

• • Setup meta_run_id and filename of the output file
  • Get the last load from table STG.incremetnal_load, control by the variable l_VAR_eBatch_PF1_QAPP_INC_LOAD and configuration the logic of the incremental load in tJava to use the date from incremental_load to the field of create or change date in the SAP
  • Call the detail job and pass parameters such as user/password, query from point number 2 to do the incremental load and save the file to GCS
  • Call the standard job to upload the files from GCS to ODS
  • If the loading is OK and parameter l_VAR_heliux_[table_name]_reload = incremental, update the time on the table incremental_load. If the value is not incremental, it is the reloading
  • If everything is OK, update the log. 

Access rights

Source

• https://solvayagile.atlassian.net/rest/api/2/search 

Format

• JSON

Destination

Location

The data will keep in

• Bucket = cs-ew1-prj-data-dm-dt-[dev]-staging/xxx
• DataOean GCP = prj-data-dm-dt-[env]
• STG Table name =  prj-data-dm-dt-[env].STG.STG_JIR_0000_0000_F001_I_H_JiraSearch
• ODS Table name =  prj-data-dm-dt-[Env].ODS.ODS_JIR_0000_F001_I_H_JiraSearch
• DPL View name  = To be updated 

To save the data into GCP Industrial Data Ocean, service account is required

Format

• columnar format

Sizing

Assessment

How to validate that the generated output is valid: 

Loading

1.1 Incremental Load

1.2 Full load

1.3. Reloading data

1.4 Plan to schedule

1.5 Timing

The average time expected for  loading:

Criticality

High/Medium/Low

Logging