This document provides an overview of the two main Google Cloud Platform (GCP) services leveraged in this project: BigQuery and Dataplex. These services work together to enable efficient data storage, processing, and quality monitoring.
The project structure within GCP has been organized to ensure clear separation between domains and environments, while supporting an efficient data ingestion and data quality validation process.
For each domain within the Data Quality Monitoring Tool (DQMT), a dedicated GCP project exists for every environment:
Development (dev)
Testing (test)
Pre-Production (ppd)
Production (prod)
Each project serves as the location where:
| Domain\Environment | Development | Testing | Pre-Production | Production |
|---|---|---|---|---|
| Human Resources | prj-data-dm-hr-dev | prj-data-dm-hr-test | prj-data-dm-hr-ppd | prj-data-dm-hr-prod |
| Structure & Shared | prj-data-dm-structure-dev | prj-data-dm-structure-test | prj-data-dm-structure-ppd | prj-data-dm-structure-prod |
| Finance | prj-data-dm-finance-dev | prj-data-dm-finance-test | prj-data-dm-finance-ppd | prj-data-dm-finance-prod |
| Marketing | prj-data-dm-marketing-dev | prj-data-dm-marketing-test | prj-data-dm-marketing-ppd | prj-data-dm-marketing-prod |
| Procurement | prj-data-dm-procurement-dev | prj-data-dm-procurement-test | prj-data-dm-procurement-ppd | prj-data-dm-procurement-prod |
In addition to the domain-specific projects, a separate set of projects is used to:
Import the final views generated by the domain projects.
Define and execute data quality rules through Dataplex.
| Environment | Project |
|---|---|
| Development | prj-data-dq-selfservice-dev |
| Testing | prj-data-dq-selfservice-test |
| Pre-Production | prj-data-dq-selfservice-ppd |
| Production | prj-data-dq-selfservice-prod |
The successful implementation and maintenance of the DQMT solution is supported by a dedicated team, each contributing with specialized skills across different areas of the project:
| Name | Role | Scope |
|---|---|---|
| Ahmed Elsayed | Data Architect | Architecture and design of data pipelines and models |
| Maria João Pimenta | Data Engineer | Data ingestion, transformation, and automation |
| Ram Atirajyam | Data Engineer | Data ingestion, transformation, and automation |
| Ibrahim Mansey | Visualization Engineer | Data visualization and dashboard development |
| Mohamed Hazem | Visualization Engineer | Data visualization and dashboard development |
| Rawan Shehab | Functional Analyst | Business analysis and functional requirements |
The following Google Groups have access to the DQMT GCP projects, organized by role:
| Group | Purpose | |
|---|---|---|
| Data Architects Group | Access for Data Architects | gcp-da-prj-data-dq-selfservice-nonprod@solvay.com |
| Data Engineers Group | Access for Data Engineers | gcp-de-prj-data-dq-selfservice-nonprod@solvay.com |
| Data Analysts / Business Analysts | No specific group | — |
| Data Developers Group | Access for Data Developers | gcp-dv-prj-data-dq-selfservice@solvay.com |
Note: No new Google Groups were specifically created for this project.
The following Service Accounts are used within the DQMT project for process automation and integration:
| Service Account | Description |
|---|---|
| sbs-is-appli-qlikview.support@solvay.com | QlikView integration and support |
| sa-talend@prj-data-dq-selfservice-dev.iam.gserviceaccount.com | Talend jobs execution |
| sa-cloudfunction@prj-data-dq-selfservice-dev.iam.gserviceaccount.com | Cloud Functions automation |