...

1. Connect to the source system API by reading context from flow job
2. Setup loop to get the data
   1. tSetGlobalVar : to set the maximum number of records to read each time and set the variable nb to check when to exit the loop (start with 0)
   2. tLoop : setup the condition to exit the loop when variable nb < 0
   3. tJava: setup the offset of records in order to get new records of each loop
3. To get data from the source by using start row number from "nb" and max row number from "limit".  It read schema from the source(meta data)
4. Generate output file and save to DATA\DEV\DATA_OCEAN_DOMAIN_DT\Tmp
5. Update the offset number "nb" = "nb" + "limit"
6. Update "nb" = -1 when ((Integer)globalMap.get("tReplace_1_NB_LINE"))<= 0  in order to exit the loop
7. Upload the files all the folder( cs-ew1-prj-data-dm-dt-[dev]-staging)
8. Delete all the files in the folder (point number 5)

• Data Anonymization 

The Talend process extracts all the cases from the ITSM source, so and as part of Data Anonymization data anonymization, we have anonymized the "Human-Resources" case data.

Below are the is a list of columns which that are anonymized as part of the during ingestion. The user Users will not be able to see view the original data and , as it will be appearing appear as "*** Anonymized info ***." , as we are anonymizing as part of the ingestion process so By anonymizing at the ingestion stage, the original data doesn't get is not stored in the Cloud Storage / BQ Tables. or BigQuery tables.

• Description
• Dynamic_data_audit_info
• Dynamic_data_definitionid
• Dynamic_data_parameter
• Summary

...