Billing
- All resources should include labels for better classification and cost attribution. The labels should include but not be limited to the following:
- Org - the Syensqo organization/GBU that "owns" the resource
- Project - the project/initiative that the resource supports
- App - the application/solution that the resource supports
- All GCP projects should have budgets defined and alert notifications set on those budgets.
- The budget set for the project should be the median consumption cost of the project over the last 12 months, or the target budget estimated by the project/initiative team.
- Alerts should be set for the 50%, 75%, and 90% or budget thresholds.
- Billing data should be automatically exported to a designated BigQuery table so it can be shared with the appropriate members of management for reporting and cost optimization purposes.
Metrics
The following (non-exhaustive) list of metrics should be captured:
- Files uploaded to Cloud Storage buckets
- Files deleted from Cloud Storage buckets
- Number of rows in each BigQuery/CloudSQL table
- Size of data stored in each BigQuery/CloudSQL table
- Execution time of data processing jobs
- Number of successful data processing job executions
- Number of failed data processing job executions
- Number of successful code builds
- Number of failed code builds
KPIs
The following (non-exhaustive) list of KPIs should be calculated and available to include/visualize in reports:
- Number of files uploaded to Cloud Storage per day
- Number of files uploaded to Cloud Storage per day per lab
- Number of files deleted Cloud Storage per day
- Average execution time of all data processing jobs
- 95% execution time of all data processing jobs
- Average execution time of each data processing job
- 95% execution time of each data processing job
- Number of successful code builds per day
- Number of failed code builds per day
- Percent of successful code builds over all code build attempts, per day
- Percent of successful code builds over all code build attempts, per user
- Percent of successful code builds over all code build attempts, per user, per week
- Percent of failed code builds over all code build attempts, per user
- Percent of failed code builds over all code build attempts, per user, per week