You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

High-Level Design architecture (HLD)

Link.


Low-Level Design architecture (LLD)

Link.

Architecture Data Flow

Here is a suggested template for Data Model + Data Mapping :

DA&DA - Domain Mapping DT


DataPrep Flow

Schema showing the different STEPS of the application flow - with the data involved at each step

According to Data Ocean Blueprint

Accolade --> GCP Data Ocean DT (prj-data-dm-dt-[env])  --> GCP Product PMO (prj-data-pmo-dash-[env]) --> Output Tableau


Conceptual Data Model


Steps descriptions

Data lineage can be found in the Google Sheet here below

Note: Table on project prj-data-dm-industrial-[env]

DataSource 1 - Accolade DB

Description

BW is the application where from where we collect information. The data is extracted using specific BW queries 

Tools

Talend to collect the data from the source and store on GCP/Google Big Query.

Access rights

You can use Xtract server (ACEW1DTLNDENG02) to manually extract and monitor.

Only DA&AI DataEng's and Data Architects can access data on GCP/Google Big Query.

Source

Location

Xtract server (ACEW1DTLNDENG02)

Format

BW queries extracted via CSV file

Destination

Location

Extracted data will be stored on GCP/Cloud storage and Google Big Query.

Format

The format of the data saved in the databank

Sizing

Expected data volume for :

  • full process from source to staging (as of 6 Dec 2023)

  • incremental process from ODS to DM (as of 6 Dec 2023)

Assessment

Check the log tables in GCP on table log_tables and run_jobs to check that there is no error loading from source to staging/ods

Check the surrogate key must be unique in the data mart layer

Scheduling

Is there an automatic schedule ? Yes

At what frequency ? We have two Talend plans in the TMC:

  • PL_MECANO_DASH : Every monday at 4am (Paris time)
  • PL_MECANO_DAILY_LOAD : Every tuesday, wednesday, thursday, friday at 4 am (Paris time)

What is the trigger ? TMC

Timing

The average time expected for :

  • PL_MECANO_DASH : around 40 minutes
  • PL_MECANO_DAILY_LOAD  : less than 5 minutes

Criticality

Medium

Logging

Table table log_tables, run_jobs, log_files, and reject_files in `prj-data-dm-industrial-[environment].STG.[table]`


Draft (Notes) for DataOps Understanding:

prj-data-dm-dt :

Source →  GCP Buckets -->  prj-data-dm-dt.STG. -->  prj-data-dm-dt.ODS --> prj-data-dm-dt.DM . -->  prj-data-dm-dt.ds_pmo_dashboard.  | 
|| 
\/

Reporting will be done from PRJ-DATA-MAINTENANCE-DASH-DEV project on  DPL dataset (prj-data-pmo-dash-dev.DPL).

Tableau --> prj-data-pmo-dash-dev.DPL   --> Prj-data-pmo-dash-dev.DataOcean  -->   prj-data-dm-dt.ds_pmo_dashboard.



  • No labels