The job shows how to read data from a REST API. The API response is converted from JSON into a CSV file which is uploaded to Google Cloud Storage.
JOB DESCRIPTION
1 - The job calls the API endpoint using the tRestClient component (you will have to adapt it to your requirements as you might need to add specific headers or authentication).
2 - The json response is parsed to extract the necessary information and saved it locally as a CSV file .
3 - The CSV is then uploaded into the desired Google Cloud Storage bucket.
4 - The temporary file is deleted.
HOW TO USE THE JOB
The job can be used a skeleton to build your job. So you can :
- Copy the job from the DATA_OCEAN project and paste it into your project
- Modified configuration of its components:
- Adapt the tRestClient with your specific URL, header, authentication
- Adapt the tJSONExtractField compoent to properly parse the JSON response
- Once the job is configured you can drag and drop it into your flow
- Provide the necessary parameters to make the job work.
The parameters that you have to provide are highlighted here below (you might need to add more depending on your use case).
| CONTEXT VARIABLE | DESCRIPTION |
|---|---|
| bucketName | The name of the bucket where the file will be upload |
| bucketFolder | Bucket folder where the file will be stored. If you want to save the file at the root of the bucket, use "" . |
| fileName | Name the CSV file the will be uploaded in GCS |
| outputDirectory | Directory where the temporary file will be stored on the remote engine |
| user | The Oracle user to be used |