The components for SySight-dev-emea

We have a development, test and production environments. At least production environments exists in all three SyWay regions, which are France Central (Global, EU data), US Gov (regulated US data) and China (China regulated data).

All environments are similar with the exception that

• Dev allows access to the KeyVault from the development workplaces
• US and China contain regulated data only. Non-regulated data is put into the West Europe region also and the Fabric US and China have transparent access to the global data.

Network architecture of SySight-dev-emea

Component architecture

The Data Platform SySight is a service layer and as such it has multiple inputs and APIs to consume its services.

• Data Ingestion either happens as pull or push, depending on the source capabilities. For example data is pulled from a database by executing SQL statements at a fixed schedule. A source like Salesforce has the capability to push changes the instant it happened.
• All ingested data is put into Apache Kafka as distribution method.
• The writer for the Application Zone is collects the data and materializes all in a Fabric Lakehouse storage.
• Other processes transform the data into higher quality level data into the Platform Zone.
• And other processes transform the data into Star Schema data models persisted in the Enterprise Zone.
• This data can then be consumed via different APIs, depending on the consumers requirement and preference.
  • The SQL Endpoint for example is ideal to query the data for reports and dashboards.
  • The GraphQL Endpoint is suited for querying rows and their related data.
  • AI tools will prefer an MCP Endpoint as this protocol is natively implemented in all LLMs.
• Data always comes in a duality with metadata, commonly referenced as Analytical Data Governance. For that reason the data producers are responsible to provide the metadata, it is stored in Kafka and different APIs allow access to it.
• Additional technical components are required for internal reasons, e.g. a KeyVault to store the source system credentials in those cases where RBACs cannot be used.

In the sub chapters these parts are looked at in more detail and how things might be connected in addition.