High Level Design architecture (HLD)
Link.
Low Level Design architecture (LLD)
Link.
Architecture Data Flow
Here is a suggested template for Data Model + Data Mapping :
https://docs.google.com/spreadsheets/d/1bD8AIgsNUI2sgANoOEKTuHBlkxhsNVTD8cmOPYEloLw
DataPrep Flow
Schema showing the different STEPS of the application flow - with the data involved at each step
Steps descriptions
Describe the data and process involved at each step
DataSource 1 - Accolade DB
Description
Accolade is the application where users (mainly project managers) share data (status, forecast as examples) related to a project.
Tools
Talend to collect the data from the source and store on GCP/Google Big Query.
Access rights
Only authorized users can connect into Accolade application.
Only DA&AI DataEng's and Data Architects can access data on GCP/Google Big Query.
Source
Location
Where is this data collected ? Accolade DB via Talend generic account created for this purpose.
Format
The format of the source data
Destination
Location
Extracted data will be stored on GCP/Google Big Query.
Format
The format of the data saved in the databank
Sizing
Expected data volume for :
- full process
- incremental process
Assessment
How to validate that the generated output is valid
Scheduling
Is there an automatic schedule ? Yes, to be scheduled.
At what frequency ? to collect data 5 times/day.
What is the trigger ?
Timing
The average time expected for :
- 5 times/day (working days: from Monday to Friday)
- full process
- incremental process
Criticality
High/ Medium /Low
Logging
Logging location
DataSource 2
SAME QUESTIONS