1.0 Features - Mailjet gudsis contact import

This batch application was built to enable the daily gudsis contact import to Mailjet (Sub accounts) contact list (SYENSQO CONTACTS).
We have x numbers of sub accounts setup in Mailjet, and individual sub account have a dedicated contact list (SYENSQO CONTACTS) which has to be updated on daily basis.

Major features:

Daily generate reformatted file using Gudsis source file and copy to Google cloud storage.
Daily cron job running to upload CSV file from Google cloud storage and import into fixed contact list for configured sub accounts in Mailjet.

2.0 System Design - Mailjet gudsis contact import

Syensqo IS - Customer Support > MAILJET GUDSIS CONTACT DAILY IMPORT FOR SYENSQO > Mailjet Workflow Diagram.png

3.0 Development - Mailjet gudsis contact import

Source repository

Project	Repository
Gudsis File Upload to Mailjet	https://gitlab.syensqo.com/syensqo-it-infra-workplace/syensqo-it-mailjet/gudsis-file-upload-to-mailjet
Mailjet-Cleanup	https://gitlab.syensqo.com/syensqo-it-infra-workplace/syensqo-it-mailjet/mailjet-cleanup

Good to know:

static String CLOUDSTORAGE_BUCKETNAME = "gcp-cs-ew1-sqo-mailjet-p-gudsisexport-syensqo".
static String CSVFILENAME = "GUDSIS_to_MAILJET.CSV".
No need to provide credentials.json file in the source code. Default service account will be used. For reference: Google Cloud example on how to use Application Default Credentials.
VPC Connector is used during the deployment for all traffic.
To acquire fixed IP Address , refer readMe file deployment command.
All the files(Google user file, MFT/Gudsis file, SCO Upload File) will be stored in the files-backup-prod storage bucket for a
period of 31days and deleted after that.

Source Code Filters:

MFT File: Fetch users with UserStatus set as ACTIVE (to avoid duplicate entries)
Google user file: Active mailboxes

Technical Details:
Total Syensqo Sub-accounts: 27

Mailjet API call Threshold: 600 API call per minute (Exception will be thrown by Mailjet system if we exceed this limit)
Number of API Calls= Number of sub-accounts * Number of records deleted per minute * Number of API call for one email deletion
Number of API call for one email deletion:2

GET API call: To get the email's contact ID from Sub-account.
DELETE API call: To delete email from the Sub-account.

Current configuration: 500= 50 * 5 * 2

To limit the number of API calls, we have configured BatchSize and waitTime variables in property file , deleteRecordCounter flag in the program.
Cleanup Function will fetch the number of records defined in the Batchsize variable.
deleteRecordCounter value is set to 10 . For every 10 email's deletion(580 calls), program will pause for the duration defined in the waitTime
variable.

#Delete Batch size BatchSize: "40" #Program sleep/wait time in seconds waitTime: "60"

4.0 Deployment - Mailjet gudsis contact import

Cloud functions are configured in Gitlab Pipeline for easy deployment. To deploy manually, refer to Readme file in the source code.

We have 2 cloud functions for Gudsis to Mailjet import:
Gudsis Mailjet Upload - Gitlab: Link || GCP cloud function: Link
Gudsis Mailjet Cleanup - Gitlab: Link || GCP cloud function: Link

Gitlab Deployment:

Steps	Screenshots
1. Checkout the develop branch from Gitlab and and push the code changes to develop branch.
2. Gitlab pipeline is configured to deploy the develop branch automatically into the staging environment when the code changes are pushed/merged to develop branch.
3. When the code is merged from develop branch to master branch, a job is created in the Gitlab environments (Production) which need to be triggered manually to deploy the master branch in production.