The job J002_FTP_TO_GCS allows you to upload files that are available on a FTP server. You can pass a filemask as input variable in order to retrieve only files matching a certain pattern.
JOB DESCRIPTION
1 - The job will generate a temporary folder inside the local folder. This folder will hold the files downloaded from the FTP server
2 - The job will establish a connection to the specified FTP server. In case of error the job will raise an error and will stop.
3 - The job will read all the files the match and save it locally
4 - Each file will be uploaded to the specified cloud storage bucket. After upload the file will be deleted from the remote engine.
HOW TO USE THE JOB?
The job is fully parametrized so you can copy the job into your project to create an indipendent instance. Then you can drag and drop the copy you have just created into your flow and pass the necessary parameters.
The variable you must provide to make the job work are :
| CONTEXT VARIABLE | DESCRIPTION |
|---|---|
| l_VAR_LOCAL_FOLDER | It indicates the folder on the local pc or remote engine where the files will be saved temporarily |
| l_VAR_FILEMASK | A text pattern used to only get file whose name matches it (for instance *.csv will download all CSV files from the FTP server, test*.csv will get all CSV files starting with "test" ) |
| ftpHost | The host of the FTP server |
| ftpPort | The port used to connect to the FTP server |
| ftpUser | The username used to connect to the FTP server |
| ftpPassword | ftpUser's password |
| ftpFolder | The folder, in the FTP server,to get files from |