Data Import
Dynasys is daily refreshed from BW and SalesForce.
A global and unique process (common to all instances of one company) allows to manage files provided by those two applications. The goal is to :
- select or exclude some csv files
- make data compatible with our Unicode Standard
- import csv files in the Datahub (TEMP tables)
- force basic data in those tables
- apply defined controls to strategic data tables, if necessary revert to a previous snapshot of data
Results of import flows are centralized into DYS_GLOBAL_IO_FLOWS table, filtered on Flow_type = 'DYN_IN' and Flow_Sub_Type = 'Import_Files' and Activated = '1'
NB : Flow_Table = name of the inbouf file without its extension
Here's the algorithm dedicated to inbound csv files :
Read files
- "Files creation dt" check
If current day = Monday, the file must have been generated on Day-1 at the latest
Otherwise, the creation date of the file must be Day 0.
- "File to exclude" check
The setting file D:\DynaSys\Batchs\Settings\dscp_settings.ini on the backend server contains a list of files to exclude.
If the file is part of this list, it won't be processed.
Status = ?
Once the file is uploaded into its TEMP table, then the related line of DYS_GLOBAL_IO_FLOWS table is updated :
- status = 1
- original_Date = creation date of the file
- Last_exec = current date
If the file is defined in the list of excluded files, then the related line of DYS_GLOBAL_IO_FLOWS table is updated :
- status = 0
- Last_exec = current date
Rework data
This process allows to force some values in TEMP tables or remove lines that are not consistent.
For example :
- Force null values to zero
- Remove line / force value to zero when a numeric field contains a non-numeric value
- Remove lines for which one field contains a specific character
Apply integrity controls
Integrity controls are defined into DYS_GLOBAL_IO_FLOWS table, filtered on Flow_type = 'DYN_IN_IC' and Activated = '1'
Several types of integrity controls exist.
