Onboarding process

Onboarding Process

Data Connectivity

High Level Overview

A script or program is provided and installed in the customer's designated system or machine. This script will connect locally to the customer's select data source, extract the data values and send them over to the SolvExtract platform via an API.

Nonetheless, there are other instance when the extraction method is hosted and managed solely on the Syensqo's side.

Extraction Methods

The customer must provide information on how the data will be shared. Currently, the digital mining team supports the following options as data sources:

PI server
Excel spreadsheet(s) located in one or multiple network drives
File uploaded to a customer owned Shared Google Drive
File shared as an attachment via e-mail
File uploaded to a Syensqo owned GCP bucket

PI Method

Requirements

If the selected data source is either PI server or Excel spreadsheets, then a script will be provided and thus the customer must open firewalls and ports to the following:

The user must install the following libraries in their system where the script is deployed:

Setup

A script will be provided and should be deployed in a machine capable of accessing the data using either of the below proposed methods.
The script must be put into a cron job using task scheduler(windows) thus it runs at least once per day.

Process

When the script is kicked off, the App file will load the solvextract.yml. This file contains all the configuration needed to run the script such as PI server name, the database, credentials (if needed), the time zone, the variables and tags that need to be extracted.
The App file will then call the Solvextract file, which will be in charge of assigning the configuration values to variables, calling the PI Class to connect to the PI server and run the query, creating the connection to Google Cloud Platform, calling the Extract API class to send over the data.
The PI Extract file is in charge of connecting to the PI system locally (via localhost or using a server name or IP), running a query using the kpis tags, getting the values back and storing them in a variable.
When the PI Extract completes its task, the Solvextract File will then create the connection to the Google Cloud Platform using a service account json key file using the GCP Bucket Class.
The GCP bucket will connect to a Cloud Storage directory where the Insert APIs key and secret are stored. These credentials are unique per customer/client.
Then the Insert API class will first call the OAth API using the credentials in order to obtain a secure token that expires after 3 minutes.
Once the token is successfully obtained, the Insert API class will call another API using the token along with a json payload that contains the extracted tag values.
After the API is invoked, the Solvay API management will redirect the API to another internal API which will validate the token and then insert the data accordingly into the Solvextract Platform.
And finally, the script log file is uploaded to the cloud storage and PI, API and Google Cloud Platform connections are closed afterwards.

Excel Extraction Method

If the selected data source is either PI server or Excel spreadsheets, then a script will be provided and thus the customer must open firewalls and ports to the following:

The customer must also provided:

Name of the file(s)
Full network path(s) where the file(s) are located.

Data Collection

Navigation tree

Expand all Collapse all

Page tree