<Provide all the ETL Jobs details, Mapping & complex Transformations
ETL jobs loading to Google Cloud Storage (GCS)
Detailed flow of ETL's from GCS to STG
Detailed flow of ETL's from STG --> ODS
Detailed flow of ETL's from ODS --> DataOcean
Detailed flow of ETL's from DataOcean --> DM
Complex Views and logic
Views expected to be used by Business/Reporting tools>
Job Details
<Include all the Flows(with screenshots) & the respective Jobs and also description of the job like the purpose of job>
For example as below:
| FlowName | Tasks | Description | Source_table/query | Target_Table | Complex transformation, if Yes,then logic |
| F001_* | J001_* J002_* J003_* | <job description> | <source table details> | <target table details> | |
| F100_* | J100_* J101_* J103_* | <job description> | <source table details> | <target table details> | |
| F200_* | J200_* J201_* J202_* | <job description> | <source table details> | <target table details> |
Data Validation/Troubleshoot queries
Data Validation:
<Provide the SQL queries to validate the data or the record count in the BQ target tables>
Logging:
<Details about the log tables in Big query>
Troubleshoot steps:
<Provide the steps to debug the ETL flow in case of failure. For example, if an ETL job fails, how should we trigger it? Can we directly rerun the job, or are there steps to delete the data from the previous execution before rerunning the job? or any change should be made in the context table for date executions>
SSL Certificates
<List out if any SSL source certificates to be added in the server with the contact details>