Data Import

Dynasys is daily refreshed from BW and SalesForce.

A global and unique process (common to all instances of one company) allows to manage files provided by those two applications. The goal is to :

• select or exclude some csv files
• make data compatible with our Unicode Standard
• import csv files in the Datahub (TEMP tables)
• force basic data in those tables
• apply defined controls to strategic data tables, if necessary revert to a previous snapshot of data

Results of import flows are centralized into DYS_GLOBAL_IO_FLOWS table,  filtered on Flow_type = 'DYN_IN' and Flow_Sub_Type = 'Import_Files' and Activated = '1'

NB : Flow_Table = name of the inbouf file without its extension

Here's the algorithm dedicated to inbound csv files :

Read files

• "Files creation dt" check

If current day = Monday, the file must have been generated on Day-1 at the latest

Otherwise, the creation date of the file must be Day 0.

• "File to exclude" check

The setting file D:\DynaSys\Batchs\Settings\dscp_settings.ini on the backend server contains a list of files to exclude.

If the file is part of this list, it won't be processed.

Status = ?

Once the file is uploaded into its TEMP table, then the related line of DYS_GLOBAL_IO_FLOWS table is updated :

• status = 1
• original_Date = creation date of the file
• Last_exec = current date

If the file is defined in the list of excluded files, then the related line of DYS_GLOBAL_IO_FLOWS table is updated :

• status = 0
• Last_exec = current date

Rework data

This process allows to force some values in TEMP tables or remove lines that are not consistent.

For example :

• Force null values to zero
• Remove line / force value to zero when a numeric field contains a non-numeric value
• Remove lines for which one field contains a specific character

Apply integrity controls

Integrity controls are defined into DYS_GLOBAL_IO_FLOWS table,  filtered on Flow_type = 'DYN_IN_IC' and Activated = '1'

Several types of integrity controls exist.