Imputation:

Before moving on to the data encoding stage, we first impute the missing values in the features using a dictionary variable in Dataiku variables, which is used by the "simple_imput()" function in the Python encoding recipe.

The details of the imputation method for each feature with missing values can be summarized in this table:

GBU	Feature	Imputation method	Details
CS	product_group	most frequent	applied for all families
CS	end_use	most frequent	applied for all families
CS	gbu_customer_seg	most frequent	applied for all families
CS	manual_region	most frequent	applied for all families
CS	lip2	most frequent	applied only for Alkoxylates
CS	historical_unit_price_coalesce_ratio_on_12	constant	value equal to 1 for all families
CS	historical_sales_coalesce_ratio_on_12	constant	value equal to 1 for all families
CS	historical_unit_price_ratio_3_on_12_month	constant	value equal to 1 for all families
CS	n_competitors	constant	applied for Sulfosuccinate_Sulfosuccinamate & Sulfosuccinates_Healthcare value equal to 1
CS	COMPONENT_ratio	mean	applied for all families
CS	IMPURITY_ratio	mean	applied for all families
CS	SOLVENT_ratio	mean	applied for all families
CS	n_components	mean	applied for all families

To monitor the percentage of imputations per feature and per CPC, we added a specific zone in the flow named "Imputation_analysis":

Pricing Front End - 11240 > 4. Data encoding > image-2024-1-11_10-0-48.png

In this zone we created three separate datasets, as follow:

Imputed_features_by_cpc: Contains imputed values (based on the adopted method defined above) for each CPC for the included features in the corresponding family. Two metrics are controlled in this dataset, the first being "tot_features_weights" which computes the impact of the imputed features for each CPC based on the average weights (from the first model) of these features. The second is "pct_imputed_cols", which calculates the percentage of imputed features in relation to the features included in the corresponding CPC family.

==> The checks carried out in this dataset compare these metrics with a fixed threshold of 0.4 for the first metric and 0.5 for the second (these values can be modified in the check parameters).

Imputed_HBs_by_cpc: Contains imputation statistics only for the features included as hard boundaries (number and percentage of imputed hard boundaries for each family by CPC).

==> The check sends a warning or an error (depending on the type of message defined in the dataiku variables) if the "pct_imputed_cols" is greater than a threshold (set to 0.5 in the check parameters).

Imputation_pct_by_feature: Calculates the percentage of imputation for each features (including HBs) by family.

==> The check verifies whether the imputation pct for a given feature is greater than the defined threshold (set at 0.5 in the check parameters).

Collapse categorical:

In the Python encoding recipe we also collapse categorical features using the "collapse_categorical()" function, in which we define a threshold to collapse modalities that have fewer values than this threshold (the default value is 5).