Description

Tools: Talend

Detail job

• J070_Kadiska_to_GCS

1. Define the date
   1. Time to select  "Begin" to "End"
   2. Time to stop "Stop"
   3. Time today = "Now"
2. Login to Kadiska API on URL: https://app.kadiska.com/api/v1/config/clients/client:5b77758db8/tokens (l_VAR_kadiska_url_token)and keep the token as global variable
3. Start looping 
4. Extract data from Kadiska API on URL: https://app.kadiska.com/api/v1/query (l_VAR_kadiska_url_query) controlling by following variables
   1. l_VAR_kadiska_q_condition : normally it is blank but it can add additional filter such as   ["=", "watcher_name", ["$","Google Calendar"]],
   2. l_VAR_kadiska_q_interval : during project design, we agree to have interval 1 hour = 3600000 milliseconds
   3. l_VAR_kadiska_q_limit : max number of records to get from API each call
   4. l_VAR_kadiska_q_nloop : max number of loop
   5. l_VAR_kadiska_q_offset : start index of the record from API query
   6. l_VAR_kadiska_q_reload : "incremental" for the incremental load, in case of reload change this variable to end time with epoch format (starting time will get from table incremental_loading on meta_file_name = KADISKA_RUM_RELOAD
5. In case [no more data] there is no data or the end time >= stop time, stop the loop
6. In case [there is data more than limit] there is data and number of line = limit, add offset + number of line
7. In case [no more data during the interval, go to next hour]there is data and number of line < limit  or there is no data and End time < Stop,  set offset = 0 and add Begin and End time 1 hour (interval)

Flow job

• F070_Kadiska_rum_to_ODS

Access rights

Source

Format

Destination

Location

Format

Sizing

Assessment

Loading

1.1 Incremental Load

1.2 Full load

1.3. Reloading data

1.4 Plan to schedule

1.5 Timing

The average time expected for  loading:

Criticality

High/Medium/Low

Logging