This job is similar to J010_MES_Unified_API_to_GCS. It adds a logic to manage incremental loading of tags using the interpolated API endpoint. The same logics can be used to extract data using a different retrieval method (e.g. the aggregates one)

PREREQUISITES

  • You should have an SAP API Management application created (in order to have an application key and application secret)
  • A"MES_control_table" must be present in your Big Query project. The table contains the last extracted timestamp for each tag.

FLOW DESCRIPTION

  1. First the job call the GenerateTokenEndpoint to retrieve an access token for the session.
  2. The job involves fetching the most recent extracted timestamp for each tag from the MES_control table within Big Query. This most recent extracted timestamp is utilized as the initial timestamp for the retrieval procedure. Subsequently, an HTTP call is initiated for each tag to the interpolated API endpoint, and the outcomes are added to a temporary CSV file.
  3. With the previously generated file, a new file is generated which encompasses the updated maximum last extracted timestamp.
  4. The two CSV files are both uploaded to Google Cloud Storage.

HOW TO USE THIS JOB


  • No labels