Deployment

You are not logged in. Any changes you make will be marked as anonymous. You may want to Log In if you already have an account.

<h1>Deployment</h1><h3 style="text-align: left;">1. Project Overview</h3>Purpose and Scope: This deployment setup provisions a containerized MQTT server and subscriber application for Freeport FMI, supporting development, pre-production, and production environments. It orchestrates the Mosquitto MQTT broker and a Python-based subscriber (which bridges MQTT to Google Cloud Pub/Sub and BigQuery) using Docker and Supervisor, and deploys the stack to Google Cloud Platform (GCP) using flexible environment configurations.Primary Use Cases:<ul style="text-align: left;"><li>Reliable ingestion of sensor data via MQTT.</li><li>Forwarding of sensor data to cloud analytics and event processing pipelines.</li><li>Environment-specific deployments (DEV, PPRD, PRD) with tailored resources and credentials.</li></ul>Explicitly Not Handled:<ul style="text-align: left;"><li>Automated scaling beyond a single instance per environment.</li><li>Secure, production-grade MQTT (no TLS/mTLS or authentication in current config).</li><li>Downstream analytics or visualization (handled elsewhere).</li></ul><hr style="text-align: left;" /><h3 style="text-align: left;">2. System Architecture</h3>Core Components and Responsibilities:<ul style="text-align: left;"><li>Docker Container: Bundles Mosquitto broker, Python subscriber, and Supervisor for process management.</li><li>Mosquitto Broker: Handles MQTT connections and message routing.</li><li>Python Subscriber: Consumes MQTT messages, publishes to Pub/Sub, and writes to BigQuery.</li><li>Supervisor: Ensures both Mosquitto and the subscriber run reliably within the container.</li><li>GCP Flexible Environment: Hosts the container, manages networking, scaling, and health checks.</li></ul>Data and Control Flow:<ol style="text-align: left;"><li>MQTT clients publish sensor data to Mosquitto broker (port 1883).</li><li>Python subscriber listens to MQTT topics, processes messages, and forwards them to Pub/Sub and BigQuery.</li><li>GCP manages container lifecycle, health checks, and network routing.</li></ol>External Services and Dependencies:<ul style="text-align: left;"><li>Google Cloud Pub/Sub (for event streaming).</li><li>Google BigQuery (for data storage).</li><li>GCP VPC and IAM (for secure networking and service account access).</li></ul><hr style="text-align: left;" /><h3 style="text-align: left;">3. Core Concepts &amp; Domain Logic</h3>Key Abstractions and Domain Terms:<ul style="text-align: left;"><li>Environment (DEV/PPRD/PRD): Drives configuration, credentials, and resource allocation.</li><li>Supervisor: Manages multiple processes inside the container.</li><li>Service Account: Used for secure access to GCP APIs.</li></ul>Business/Technical Invariants:<ul style="text-align: left;"><li>Each environment uses its own GCP project, service account, and VPC connector.</li><li>Only one instance runs per environment (no horizontal scaling).</li></ul>Mental Model:<ul style="text-align: left;"><li>Each deployment is a self-contained MQTT ingestion node, tightly integrated with GCP services and isolated per environment.</li></ul><hr style="text-align: left;" /><h3 style="text-align: left;">4. Codebase Structure</h3>High-level Layout:<ul style="text-align: left;"><li><code class="undefined">mqtt_subscriber.py</code>: Python application for MQTT-to-cloud bridging.</li><li><code class="undefined">mosquitto.conf</code>: Mosquitto broker configuration.</li><li><code class="undefined">supervisord.conf</code>: Supervisor configuration for process management.</li><li><code class="undefined">requirements.txt</code>: Python dependencies.</li><li>Dockerfile: Container build instructions.</li><li>Environment-specific YAMLs (<code class="undefined">dev.yml</code>, <code class="undefined">pprd.yml</code>, <code class="undefined">prd.yml</code>): GCP deployment descriptors.</li><li>CI/CD YAML (<code class="undefined">project-ci.yml</code>): Pipeline and deployment automation.</li></ul>Responsibility Boundaries:<ul style="text-align: left;"><li>Application logic (Python) vs. infrastructure (Docker, Mosquitto, Supervisor).</li><li>Environment-specific configuration is isolated in separate YAML files.</li></ul>What Changes Together:<ul style="text-align: left;"><li>Application code and requirements.</li><li>Mosquitto and Supervisor configs.</li><li>Environment YAMLs and CI/CD pipeline.</li></ul><hr style="text-align: left;" /><h3 style="text-align: left;">5. Configuration &amp; Environment</h3>Environment Variables:<ul style="text-align: left;"><li>Set in deployment YAMLs (e.g., <code class="undefined">APP_ENV</code>, <code class="undefined">GCP_PROJECT_ID</code>, <code class="undefined">MQTT_BROKER</code>, etc.).</li><li>Control application behavior and cloud integration.</li></ul>Configuration Files:<ul style="text-align: left;"><li><code class="undefined">mosquitto.conf</code>: Broker settings.</li><li><code class="undefined">supervisord.conf</code>: Process management.</li><li><code class="undefined">requirements.txt</code>: Python dependencies.</li></ul>Differences Between Local, Staging, and Production:<ul style="text-align: left;"><li>Each environment uses a different GCP project, service account, VPC connector, and resource allocation.</li><li>Production (<code class="undefined">prd.yml</code>) uses more CPU/memory than DEV/PPRD.</li></ul><hr style="text-align: left;" /><h3 style="text-align: left;">6. Runtime Behavior</h3>Startup Sequence:<ul style="text-align: left;"><li>Container starts, Supervisor launches Mosquitto and Python subscriber.</li><li>Mosquitto listens on port 1883.</li><li>Subscriber connects to broker and cloud services.</li></ul>Normal Execution Flow:<ul style="text-align: left;"><li>MQTT messages are received, processed, and forwarded to cloud.</li><li>Supervisor restarts processes if they fail.</li></ul>Error Handling and Logging Strategy:<ul style="text-align: left;"><li>Application logs to stdout (captured by GCP logging).</li><li>Mosquitto logs as configured.</li><li>Liveness/readiness checks ensure unhealthy containers are restarted.</li></ul><hr style="text-align: left;" /><h3 style="text-align: left;">7. Deployment &amp; Operations</h3>Build Process:<ul style="text-align: left;"><li>Docker image is built from the Dockerfile (Python, Mosquitto, Supervisor, app code).</li></ul>Deployment Method:<ul style="text-align: left;"><li>CI/CD pipeline (<code class="undefined">project-ci.yml</code>) deploys the container to GCP Flexible Environment using environment-specific YAMLs.</li><li>Each deployment uses a dedicated service account and VPC connector.</li></ul>Runtime Dependencies:<ul style="text-align: left;"><li>GCP credentials (service account key or workload identity).</li><li>Network access to GCP APIs.</li></ul>Scaling and Rollback Considerations:<ul style="text-align: left;"><li>Only one instance per environment (no auto-scaling).</li><li>Rollback by redeploying a previous image/tag.</li></ul><hr style="text-align: left;" /><h3 style="text-align: left;">8. Extending the System</h3>Where and How to Add New Features:<ul style="text-align: left;"><li>Update Python code for new message handling or cloud integrations.</li><li>Adjust Mosquitto or Supervisor configs as needed.</li><li>Add environment variables to YAMLs for new configuration options.</li></ul>Recommended Patterns:<ul style="text-align: left;"><li>Use environment variables for all configuration.</li><li>Keep environment YAMLs in sync with code/config changes.</li><li>Test changes in DEV before promoting to PPRD/PRD.</li></ul>Anti-patterns and Risk Areas:<ul style="text-align: left;"><li>Hardcoding secrets or credentials.</li><li>Making breaking changes to topic structure or message schema without coordination.</li><li>Running multiple instances without unique MQTT client IDs.</li></ul>Testing Strategy:<ul style="text-align: left;"><li>Unit/integration tests for Python code.</li><li>Deploy to DEV and validate end-to-end message flow before promoting.</li></ul><hr style="text-align: left;" /><h3 style="text-align: left;">9. Security &amp; Compliance</h3>Authentication and Authorization:<ul style="text-align: left;"><li>GCP access is secured via service accounts.</li><li>MQTT broker is not secured (no TLS or authentication) – must be addressed for production.</li></ul>Secrets Handling:<ul style="text-align: left;"><li>No secrets in code; all sensitive data should be managed via environment or GCP Secret Manager.</li></ul>Data Sensitivity Considerations:<ul style="text-align: left;"><li>Sensor data may be sensitive; ensure GCP IAM and network policies are enforced.</li></ul><hr style="text-align: left;" /><h3 style="text-align: left;">10. Common Pitfalls &amp; Gotchas</h3><ul style="text-align: left;"><li>No MQTT Security: Broker is open on 1883 with no authentication/TLS; not suitable for production as-is.</li><li>Single Instance: No horizontal scaling; may be a bottleneck for high-throughput scenarios.</li><li>Resource Limits: Production uses more resources, but disk is fixed at 10GB; monitor for growth.</li><li>Environment Drift: YAMLs must be kept in sync with code/config changes.</li><li>Supervisor: If Supervisor or one of its managed processes fails to start, the container will not function correctly.</li><li>Health Checks: Liveness/readiness checks are critical for auto-recovery but may need tuning for startup delays.</li></ul>