6. Data Flow Diagram

Overview

WebApp Flow

WebAppFlow.drawio

Data input from WebApp to Big query will go directly project prj-robustify-[env] in dataset DS_APP_Robustify for the tables name [site]_inputs

Then the data flow will be: 

1. DS_APP_Robutify.[site]_inputs    The data that get from WebApp
2. DM.V_FACT_[site]_inputs   Change row to column. (to change parameter name of webApp to be column in BQ)
3. DM.V_FACT_[site]_inputs_enr to add missing value if webApp does not have data, it will get from Data Ocean
4. DM.V_FACT_[site]_inputs_queue Same data as DM.V_FACT_[site]_inputs_enr but only records are not yet calculate by Dataiku. Checking by DS_APP_dataiku. [site]_optimization_status table
5. DPS_APP_dataiku.[site]_outputs/DPS_APP_dataiku.[site]_optimization_status  The data from DPL.V_FACT_[site]_inputs_queue will process in Dataiku server and send the output and update status to these table
6. DM.V_FACT_[site]_output  get data from DPS_APP_dataiku.[site]_outputs 
7. DM.V_FACT_marginal_[site]  is the join tables on DS_APP_Robutify.[site]_inputs and DPS_APP_dataiku.[site]_outputs 
8. DM.V_FACT_[site]_input_enr_Tableau  to change data from text to float for Tableau to use the input data

Remark: DM.[table] = DPL.[Table]  The data will be the same but DPL will not have any calculation and the third party like Dataiku will read the data from DPL only. 

Talend Flow

Visual representation of data flow from sources to data presentation layer.

Include arrows and labels to illustrate the flow.
White → Industrial domain

Blue → Sustainability domain

Flow.drawio