This document provides an overview of the two main Google Cloud Platform (GCP) services leveraged in this project: BigQuery and Dataplex. These services work together to enable efficient data storage, processing, and quality monitoring.
The project structure within GCP has been organized to ensure clear separation between domains and environments, while supporting an efficient data ingestion and data quality validation process.
For each domain within the Data Quality Monitoring Tool (DQMT), a dedicated GCP project exists for every environment:
Development (dev)
Testing (test)
Pre-Production (ppd)
Production (prod)
Each project serves as the location where:
Domain\Environment | Development | Testing | Pre-Production | Production |
|---|---|---|---|---|
| Human Resources | prj-data-dm-hr-dev | prj-data-dm-hr-test | prj-data-dm-hr-ppd | prj-data-dm-hr-prod |
| Structure & Shared | prj-data-dm-structure-dev | prj-data-dm-structure-test | prj-data-dm-structure-ppd | prj-data-dm-structure-prod |
| Finance | prj-data-dm-finance-dev | prj-data-dm-finance-test | prj-data-dm-finance-ppd | prj-data-dm-finance-prod |
| Marketing | prj-data-dm-marketing-dev | prj-data-dm-marketing-test | prj-data-dm-marketing-ppd | prj-data-dm-marketing-prod |
| Procurement | prj-data-dm-procurement-dev | prj-data-dm-procurement-test | prj-data-dm-procurement-ppd | prj-data-dm-procurement-prod |
In addition to the domain-specific projects, a separate set of projects is used to:
Import the final views generated by the domain projects.
Define and execute data quality rules through Dataplex.
Environment | Project |
|---|---|
| Development | prj-data-dq-selfservice-dev |
| Testing | prj-data-dq-selfservice-test |
| Pre-Production | prj-data-dq-selfservice-ppd |
| Production | prj-data-dq-selfservice-prod |
The successful implementation and maintenance of the DQMT solution is supported by a dedicated team, each contributing with specialized skills across different areas of the project:
Name | Role | Scope |
|---|---|---|
| Ahmed Elsayed | Data Architect | Architecture and design of data pipelines and models |
| Maria João Pimenta | Data Engineer | Data ingestion, transformation, and automation |
| Ram Atirajyam | Data Engineer | Data ingestion, transformation, and automation |
| Ibrahim Mansey | Visualization Engineer | Data visualization and dashboard development |
| Mohamed Hazem | Visualization Engineer | Data visualization and dashboard development |
| Rawan Shehab | Functional Analyst | Business analysis and functional requirements |
The following Google Groups have access to the DQMT GCP projects, organized by role:
Group | Purpose | |
|---|---|---|
| Data Architects Group | Access for Data Architects | gcp-da-prj-data-dq-selfservice-nonprod@solvay.com |
| Data Engineers Group | Access for Data Engineers | gcp-de-prj-data-dq-selfservice-nonprod@solvay.com |
| Data Analysts / Business Analysts | No specific group | — |
| Data Developers Group | Access for Data Developers | gcp-dv-prj-data-dq-selfservice@solvay.com |
Note: No new Google Groups were specifically created for this project.
The following Service Accounts are used within the DQMT project for process automation and integration:
Service Account | Description |
|---|---|
| sbs-is-appli-qlikview.support@solvay.com | QlikView integration and support |
| sa-talend@prj-data-dq-selfservice-dev.iam.gserviceaccount.com | Talend jobs execution |
| sa-cloudfunction@prj-data-dq-selfservice-dev.iam.gserviceaccount.com | Cloud Functions automation |
Domain Projects
Projects | View | Source Tables/Views |
| prj-data-dm-hr-[env] | DS_prj_dqkpi.V_BusinessUnit | ODS.ODS_SFC_0000_F001_F_W_FOBusinessUnit |
| DS_prj_dqkpi.V_CcHrFin | ODS.ODS_SFC_0000_F001_F_W_FOCostCenter ODS.ODS_BWH_0000_F001_F_W_BW_QRY_C_COSTCTR_0001 | |
| DS_prj_dqkpi.V_Company | ODS.ODS_SFC_0000_F001_F_W_FOCompany | |
| DS_prj_dqkpi.V_CostCenter | ODS.ODS_SFC_0000_F001_F_W_FOCostCenter | |
| DS_prj_dqkpi.V_EmpBusiness | DS_prj_dqkpi.V_EmpJob DS_prj_dqkpi.V_BusinessUnit | |
| DS_prj_dqkpi.V_EmpCompensation | ODS.ODS_SFC_0000_F001_F_W_EmpCompensation DS_prj_dqkpi.V_PickListValueV2 | |
| DS_prj_dqkpi.V_EmpCompensationJob | DS_prj_dqkpi.V_EmpCompensation DS_prj_dqkpi.V_EmpJob | |
| DS_prj_dqkpi.V_EmpJob | DS_prj_dqkpi.V_PickListValueV2 DS_prj_dqkpi.V_FOLocation DS_prj_dqkpi.V_EmpJobCoded | |
| DS_prj_dqkpi.V_EmpJobCoded | ODS.ODS_SFC_0000_F001_F_D_EmpJob | |
| DS_prj_dqkpi.V_EmpJob_CC | DS_prj_dqkpi.V_EmpJob DS_prj_dqkpi.V_CostCenter | |
| DS_prj_dqkpi.V_EmpJob_CompensationGroupSumCalc | ODS.ODS_SFC_0000_F001_F_W_empCompensationGroupSumCalculatedNav DS_prj_dqkpi.V_EmpJob | |
| DS_prj_dqkpi.V_EmpJob_LocGroup | DS_prj_dqkpi.V_EmpJob DS_prj_dqkpi.V_LocationGroup | |
| DS_prj_dqkpi.V_EmpJob_Location | DS_prj_dqkpi.V_EmpJob DS_prj_dqkpi.V_FOLocation | |
| DS_prj_dqkpi.V_FOLocation | ODS.ODS_SFC_0000_F001_F_W_FOLocation | |
| DS_prj_dqkpi.V_LocationGroup | ODS.ODS_SFC_0000_F001_F_W_FOLocationGroup | |
| DS_prj_dqkpi.V_PickListValueV2 | ODS.ODS_SFC_0000_F001_F_W_PickListValueV2 | |
| DS_prj_dqkpi.V_Position | ODS.ODS_SFC_0000_F001_F_W_Position | |
| DS_prj_dqkpi.V_PositionJobInfo | DS_prj_dqkpi.V_EmpJob | |
| prj-data-dm-structure-[env] | DS_prj_sls_dataquality_kpi.V_PF1 | ODS.ODS_SPF_0000_F001_F_W_T001W |
| DS_prj_sls_dataquality_kpi.V_PF1_MVKE_T001W_TVKOT_T001K | ODS.ODS_SPF_0000_F001_F_W_MVKE ODS.ODS_SPF_0000_F001_F_W_T001W ODS.ODS_SPF_0000_F001_F_W_TVKOT | |
| DS_prj_sls_dataquality_kpi.V_PF1_MVKE_T001W_TVKOT_T001K_U | DS_prj_sls_dataquality_kpi.V_PF1_MVKE_T001W_TVKOT_T001K | |
| DS_prj_sls_dataquality_kpi.V_WP1 | ODS.ODS_SPW_0000_F001_F_W_T001W ODS.ODS_SPW_0000_F001_F_W_T001K | |
| DS_prj_sls_dataquality_kpi.V_WP1_MVKE_T001W_TVKOT_T001K | ODS.ODS_SPW_0000_F001_F_W_MVKE ODS.ODS_SPW_0000_F001_F_W_T001W ODS.ODS_SPW_0000_F001_F_W_TVKOT | |
| DS_prj_sls_dataquality_kpi.V_WP1_MVKE_T001W_TVKOT_T001K_U | DS_prj_sls_dataquality_kpi.V_WP1_MVKE_T001W_TVKOT_T001K | |
| prj-data-dm-finance-[env] | DS_prj_sls_dataquality_kpi.V_COSTCTR_0001 | ODS.ODS_BWH_0000_F001_F_W_BW_QRY_C_COSTCTR_0001 |
| prj-data-dm-marketing-[env] | DS_prj_sls_dataquality_kpi.V_PF1_KNA1 | ODS.ODS_SPF_0000_F001_F_W_KNA1 |
| DS_prj_sls_dataquality_kpi.V_WP1_KNA1 | ODS.ODS_SPW_0000_F001_F_W_KNA1 | |
| prj-data-dm-procurement-[env] | DS_sls_dataquality_kpi.V_PF1_MARC | ODS.ODS_SPF_0000_F001_F_W_MARC |
| DS_sls_dataquality_kpi.V_WP1_MARC | ODS.ODS_SPW_0000_F001_F_W_MARC |
Selfservice Project
Project | Table/View | Source Tables/Views |
|---|---|---|
| prj-data-dm-selfservice-[env] | DM.DIM_date | - |
DM.DIM_domain | - | |
DM.DIM_kpi_dimension | - | |
DM.DIM_quality_rule | - | |
DM.Dataplex_quality | - | |
DM.FACT_data_quality | - | |
DM.FACT_failed_records | - | |
DPL.V_DIM_DATE | prj-data-dm-selfservice-[env].DM.DIM_date | |
DPL.V_DIM_DOMAIN | prj-data-dm-selfservice-[env].DM.DIM_domain | |
DPL.V_DIM_KPI_DIMENSION | prj-data-dm-selfservice-[env].DM.DIM_kpi_dimension | |
DPL.V_FACT_QUALITY | prj-data-dm-selfservice-[env].DM.FACT_data_quality | |
DPL.V_RULE_QUALITY | prj-data-dm-selfservice-[env].DM.DIM_quality_rule | |
DataOcean_dataquality_kpi.V_ActiveEmployeeInActiveLegalEntity | prj-data-dm-hr-[env].DS_prj_dqkpi.V_EmpJob prj-data-dm-hr-[env].DS_prj_dqkpi.V_Company | |
DataOcean_dataquality_kpi.V_COSTCTR_0001 | prj-data-dm-finance-[env].DS_prj_sls_dataquality_kpi.V_COSTCTR_0001 | |
DataOcean_dataquality_kpi.V_COSTCTR_0001_ProfitCenter | prj-data-dm-finance-[env].DS_prj_sls_dataquality_kpi.V_COSTCTR_0001 | |
DataOcean_dataquality_kpi.V_EmpBusiness | prj-data-dm-hr-[env].DS_prj_dqkpi.V_EmpBusiness | |
DataOcean_dataquality_kpi.V_EmpCompensationJob | prj-data-dm-hr-[env].DS_prj_dqkpi.V_EmpCompensationJob | |
DataOcean_dataquality_kpi.V_EmpJob_CC | prj-data-dm-hr-[env].DS_prj_dqkpi.V_EmpJob_CC | |
DataOcean_dataquality_kpi.V_EmpJob_Cadres | prj-data-dm-hr-[env].DS_prj_dqkpi.V_EmpJob | |
DataOcean_dataquality_kpi.V_EmpJob_Compensation_AIS | prj-data-dm-hr-[env].DS_prj_dqkpi.V_EmpJob_CompensationGroupSumCalc | |
DataOcean_dataquality_kpi.V_EmpJob_Compensation_AIS_ANSG | prj-data-dm-hr-[env].DS_prj_dqkpi.V_EmpJob_CompensationGroupSumCalc | |
DataOcean_dataquality_kpi.V_EmpJob_Compensation_ANSG | prj-data-dm-hr-[env].DS_prj_dqkpi.V_EmpJob_CompensationGroupSumCalc | |
DataOcean_dataquality_kpi.V_EmpJob_External_Pay | prj-data-dm-hr-[env].DS_prj_dqkpi.V_EmpJob | |
DataOcean_dataquality_kpi.V_EmpJob_External_Position | prj-data-dm-hr-[env].DS_prj_dqkpi.V_EmpJob | |
DataOcean_dataquality_kpi.V_EmpJob_LocGroup | prj-data-dm-hr-[env].DS_prj_dqkpi.V_EmpJob_LocGroup | |
DataOcean_dataquality_kpi.V_EmpJob_Location | prj-data-dm-hr-[env].DS_prj_dqkpi.V_EmpJob_Location | |
DataOcean_dataquality_kpi.V_EmpJob_Supervisor | prj-data-dm-hr-[env].DS_prj_dqkpi.V_EmpJob | |
DataOcean_dataquality_kpi.V_FOLocation | prj-data-dm-hr-[env].DS_prj_dqkpi.V_FOLocation | |
DataOcean_dataquality_kpi.V_PF1 | prj-data-dm-structure-[env].DS_prj_sls_dataquality_kpi.V_PF1 | |
DataOcean_dataquality_kpi.V_PF1_KNA1 | prj-data-dm-marketing-[env].DS_prj_sls_dataquality_kpi.V_PF1_KNA1 | |
DataOcean_dataquality_kpi.V_PF1_MARC | prj-data-dq-selfservice-[env].DataOcean_dataquality_kpi.V_proc_PF1_MARC prj-data-dq-selfservice-[env].DataOcean_dataquality_kpi.V_PF1 | |
DataOcean_dataquality_kpi.V_PF1_MVKE_T001W_TVKOT_T001K_U | prj-data-dm-structure-[env].DS_prj_sls_dataquality_kpi.V_PF1_MVKE_T001W_TVKOT_T001K_U | |
DataOcean_dataquality_kpi.V_PositionJobInfo | prj-data-dm-hr-[env].DS_prj_dqkpi.V_PositionJobInfo | |
DataOcean_dataquality_kpi.V_PositionJobInfo_BU | prj-data-dm-hr-[env].DS_prj_dqkpi.V_PositionJobInfo | |
DataOcean_dataquality_kpi.V_PositionJobInfo_CC | prj-data-dm-hr-[env].DS_prj_dqkpi.V_PositionJobInfo | |
DataOcean_dataquality_kpi.V_WP1 | prj-data-dm-structure-[env].DS_prj_sls_dataquality_kpi.V_WP1 | |
DataOcean_dataquality_kpi.V_WP1_KNA1 | prj-data-dm-marketing-[env].DS_prj_sls_dataquality_kpi.V_WP1_KNA1 | |
DataOcean_dataquality_kpi.V_WP1_MARC | prj-data-dq-selfservice-[env].DataOcean_dataquality_kpi.V_proc_WP1_MARC prj-data-dq-selfservice-[env].DataOcean_dataquality_kpi.V_WP1 | |
DataOcean_dataquality_kpi.V_WP1_MVKE_T001W_TVKOT_T001K_U | prj-data-dm-structure-[env].DS_prj_sls_dataquality_kpi.V_WP1_MVKE_T001W_TVKOT_T001K_U | |
DataOcean_dataquality_kpi.V_proc_PF1_MARC | prj-data-dm-procurement-[env].DS_sls_dataquality_kpi.V_PF1_MAR | |
DataOcean_dataquality_kpi.V_proc_WP1_MARC | prj-data-dm-procurement-[env].DS_sls_dataquality_kpi.V_WP1_MARC |
Project | Routine | Output |
|---|---|---|
| prj-data-dm-selfservice-[env] | DM.insert_DIM_Domain | Populates the Data Model Dimension Table DM.DIM_domain |
| DM.insert_DIM_kpi_dimension | Populates the Data Model Dimension Table DM.DIM_kpi_dimension | |
| DM.RT_DPtoDMmapping_Datespecific | Populates the Data Model Fact Tables: |