Billing

• All resources should include labels for better classification and cost attribution. The labels should include but not be limited to the following:
  • Org - the Syensqo organization/GBU that "owns" the resource
  • Project - the project/initiative that the resource supports
  • App - the application/solution that the resource supports
• All GCP projects should have budgets defined and alert notifications set on those budgets.
  • The budget set for the project should be the median consumption cost of the project over the last 12 months, or the target budget estimated by the project/initiative team.
  • Alerts should be set for the 50%, 75%, and 90% or budget thresholds.
• Billing data should be automatically exported to a designated BigQuery table so it can be shared with the appropriate members of management for reporting and cost optimization purposes.

Metrics

The following (non-exhaustive) list of metrics should be captured:

• Files uploaded to Cloud Storage buckets
• Files deleted from Cloud Storage buckets
• Number of rows in each BigQuery/CloudSQL table
• Size of data stored in each BigQuery/CloudSQL table
• Execution time of data processing jobs
• Number of successful data processing job executions
• Number of failed data processing job executions
• Number of successful code builds
• Number of failed code builds

KPIs

The following (non-exhaustive) list of KPIs should be calculated and available to include/visualize in reports:

• Number of files uploaded to Cloud Storage per day
• Number of files uploaded to Cloud Storage per day per lab
• Number of files deleted Cloud Storage per day
• Average execution time of all data processing jobs
• 95% execution time of all data processing jobs
• Average execution time of each data processing job
• 95% execution time of each data processing job
• Number of successful code builds per day
• Number of failed code builds per day
• Percent of successful code builds over all code build attempts, per day
• Percent of successful code builds over all code build attempts, per user
• Percent of successful code builds over all code build attempts, per user, per week
• Percent of failed code builds over all code build attempts, per user
• Percent of failed code builds over all code build attempts, per user, per week