Replacing BASIC routines in DataStage

Last updated: Dec 09, 2024

To replace a BASIC routine, rewrite the routine code into a script that uses the dsjob CLI or REST API calls. Add it to your Pipeline flow as a Run Bash script node.

Authentication token

To get started with the API server on REST or CLI, get the authentication token for the server.

For more information, see Creating a CPD bearer token.

Assuming your CPD UI URL is in the format https://cpd-ds.apps.yourcompany.com, get the authentication tokens with the following code.

The iam token:

export CPD_URL=https://cpd-ds.apps.yourcompany.com
export USERNAME=<your login user>
export PASSWORD=<your login password>
iamtoken=`curl -k -s -X POST -H "Content-Type: application/x-www-form-urlencoded;charset=UTF-8" -d "grant_type=password&username=${USERNAME}&password=${PASSWORD}&scope=openid" ${CPD_URL}/idprovider/v1/auth/identitytoken | jq .access_token | cut -d\" -f2`

The ZenApi key authentication token:

curl -k -X POST -H "cache-control: no-cache" -H "Content-Type: application/json" -d "{\"username\":\"{username}\",\"api_key\": \"{api_key}\"}" "https://{cpd_cluster_host}/icp4d-api/v1/authorize"

Replace a routine

The following BASIC routine gets the name of a job:

# **      Jobname = DSGetJobInfo (DSJ.ME,DSJ.JOBNAME)

You can obtain the same information by writing a script that uses the REST API and adding it to a Run Bash Script node.

Obtain parameters such as the project id, the project name, the job id, and the job run id. You can get these values through CEL expressions: ctx.scope.id, ctx.scope.name, ctx.job.id, ctx.job_run.id, or you can do so with the REST API as demonstrated in the following commands. This example assumes the name of your project is project1.

Project ID

$ project=`curl --request GET "${CPD_URL}/v2/projects?name=project1" -H "Authorization: Bearer ${bearer_token}" -k -s`
$ proj_id=`echo $project | jq -r '.resources|.[0].metadata.guid'`

Job ID

$ joblist=`curl --request GET "${CPD_URL}/v2/jobs?project_id=${proj_id}" --header "Authorization: Bearer ${bearer_token}" -k -s`
$ job_id= `echo $joblist | jq -r '.results[].metadata|select(.name=="testjob"?) |.asset_id'`

Run ID

$ runlist=`curl --request GET "${CPD_URL}/v2/jobs/${job_id}/runs?project_id=${proj_id}" --header "Authorization: Bearer ${bearer_token}" -k -s`
$ run_id= `echo $runlist|jq -r '.results[]|select(.metadata.description=="Initial run"?) |.metadata.asset_id'`

Get the metadata for a job run in a specified project. Use the parameters proj_id, job_id, and run_id defined by the previous commands.

$ jobrun=`curl --request GET "${CPD_URL}/v2/jobs/${job_id}/runs/${run_id}?job_id=${job_id}&run_id=${run_id}&project_id=${proj_id}" --header "Authorization: Bearer ${bearer_token}"`

Extract the information.

$ echo $jobrun|jq -r '.entity.job_run.job_name'

This example uses REST API commands, but dsjob can also be used. It might be useful to rewrite routines as wrapper functions so they can be reused throughout your scripts. See Examples for more information and examples.