Aggregat - DataOps PoC Summary

Objective : Migration to DataOps practices of Aggregat project

Project Manager : Christophe AUDEFROY

Team : DIGGLERZ Factory (NODYA Group)

PoC References :

• PoC Description : 

• Documentation (in /Documentation)

  • Aggregat - 1 - Management: Project presentation, Criticality, Referees, Knowledge management

  • Aggregat - 2 - Functional: Description, Stakeholders, Constraints

  • Aggregat - 3 - Technical - Data Preparation: Data Architecture Flow, Steps description

  • Aggregat - 4 - Technical - Data Application: Data Architecture Flow, Steps description

  • Aggregat - 5 - Operational: Procedures, Scheduling, Monitoring, Error Handling

  • Aggregat - 6 - Maintenance: On-boarding

• Maintenance Roadmap kickstart : Aggregat - Roadmap

• Some generic Dataiku recommendations :

Roadmap HIGH priority steps Highlight :

1. Migrate from DataOps PoC ("TEST") to DEV environments

2. Increment Aggregat DataOps with missing evolutions

3. Integrate DataApp outputs in Energy DB

4. Go to PROD

5. PoV architecture removal

6. DataPrep (Weather - Grib) deployment improvement

7. DataApp (SES Aggregat) functionnal testing performances

Actual Projects locations / resources :

• DataPrep "Weather Data Extraction"

  • project : https://gitlab.solvay.com/solvay-it-dataops/data-ingestion/weather-data-extraction-grib/components/weather-data-extraction

• DataPrep

  • sources : https://gitlab.solvay.com/solvay-it-dataops/data-ingestion/ses-agregat-dataprep/components/ses-agregat-dataprep

  • project : https://dss-test.solvay.com/projects/SESAGREGATDATAPREP/

• DataApp

  • sources : https://gitlab.solvay.com/solvay-it-dataops/data-application/ses_agregat/components/ses_agregat

  • project : https://dss-test.solvay.com/projects/SESAGGREGAT/

  • data output : https://console.cloud.google.com/storage/browser/cs-ew1-solvay-energy-aggregation-dev-sesaggregat/dataiku/SESAGGREGAT?project=solvay-energy-aggregation-dev

• DataPrep Pipeline : https://gitlab.solvay.com/solvay-it-dataops/data-ingestion/ses-agregat-dataprep/components/ses-agregat-dataprep/-/pipelines

• DataApp Pipeline : https://gitlab.solvay.com/solvay-it-dataops/data-application/ses_agregat/components/ses_agregat/-/pipelines

• DataBank : https://console.cloud.google.com/storage/browser/cs-ew1-bda-databank-dev-dataiku?project=prj-bda-databank-dev