View Source

The job shows how to read data from a REST API. The API response is converted from JSON into a CSV file which is uploaded to Google Cloud Storage.

JOB DESCRIPTION

Data Ocean > J008_REST_API_to_GCS - Read data from a REST API > image2023-8-21_8-26-27.png

1 - The job calls the API endpoint using the tRestClient component (you will have to adapt it to your requirements as you might need to add specific headers or authentication).

2 - The json response is parsed to extract the necessary information and saved it locally as a CSV file .

3 - The CSV is then uploaded into the desired Google Cloud Storage bucket.

4 - The temporary file is deleted.

HOW TO USE THE JOB

The job can be used a skeleton to build your job. So you can :

Copy the job from the DATA_OCEAN project and paste it into your project
Modified configuration of its components:
- Adapt the tRestClient with your specific URL, header, authentication
- Adapt the tJSONExtractField compoent to properly parse the JSON response
Once the job is configured you can drag and drop it into your flow
Provide the necessary parameters to make the job work.

The parameters that you have to provide are highlighted here below (you might need to add more depending on your use case).

CONTEXT VARIABLE	DESCRIPTION

bucketName	The name of the bucket where the file will be upload
bucketFolder	Bucket folder where the file will be stored. If you want to save the file at the root of the bucket, use "" .
fileName	Name the CSV file the will be uploaded in GCS
outputDirectory	Directory where the temporary file will be stored on the remote engine
user	The Oracle user to be used