Migrating and constructing pipeline flows for DataStage

The following steps and limitations apply to migrated Sequence Jobs and flows that are constructed directly with the pipeline canvas.

Migrated flows

For more information on each component, please see Pipeline components for DataStage.

Wait for file

Manually reselect or configure the file path. As a helper node for cross loop, the default timeout value is 23:59:59. Manually update value or set to 00:00:00 for no timeout.

Wait for all

Replaces Sequencer (all) and Nested condition.

Wait for any

Replaces Sequencer (any).

Terminate pipeline

Replaces Terminator.

Final message text is not supported.

Terminate loop

Not supported during migration, must be manually added to a loop.

Loop in sequence

Replaces Start/end loop.

Run DataStage job

Replaces Job activity for parallel jobs.

The List type is mapped to Enum. Path is mapped to the File type. For information see Configuring global objects for Orchestration Pipelines.

Run Pipelines job

Replaces Job activity for sequence jobs. For information see Run Pipelines job.

Run Bash script

You must replace single quotes around environment variables with double quotes so they are not treated as string literals.

Set user variables

Replaces User variable.

Error handling

Replaces Exception handler.

Use error.status and error.status_message to get the failed node's information. Use ds.GetErrorSource() and ds.GetErrorNumber() to get the error source and error number.

General pipeline issues

Unsupported functions return "1" or "unsupported."
When outside nodes are accessed inside a loop or exception handler, migration adds an extra Set user variables node.
When the main pipeline has an Exception handler node or when a Loop node has links that point out to the main pipeline, migration creates extra nodes and a local parameter, MigrationTempFolder. A Run Bash script node is created inside the Loop or Exception handler. This node creates a linkage file under the mounted path provided in MigrationTempFolder, which must include a / at the end. Outside of the Loop or Exception handler, a Wait for file node is created which waits for the linkage file to be created. If the mounted path is incorrect the node will wait 24 hours. Another Run Bash script node deletes the linkage file afterward.
When Automatically handle activities that fail is selected, all migrated run nodes will be set to Fail on pipeline error, unless a condition has been defined on a link for when an error is thrown. If a condition has been defined or Automatically handle activities that fail is not defined, nodes will be set to Continue pipeline on error.
During migration, all parallel and sequence jobs along with all their dependencies need to be included in the ISX file. If one sequence job depends on another sequence job, but the dependent sequence job is not included in the migrated ISX file, migration marks the dependent job as Run DataStage job node instead of Run Pipeline job node. Also, migration creates extra nodes for any missing parameters.

Set and get user status

To set user status in a DataStage job, you can call the built-in function SetUserStatus from the Expression builder in the Transformer stage. When you go to Triggers in the Transformer and call SetUserStatus, it cannot be used on input column derivations.

To get the status in a pipeline that calls the DataStage job with a Run DataStage job node, you can use the built-in function ds.GetUserStatus(tasks.<node name>) with the name of the Run DataStage job node. You can also access it in the job results with tasks.<node name>.user_status. To set user status in a pipeline, you must add it as a variable with the Set user variables node and select Make user variable value available as a pipeline result, which makes it an output parameter that other pipelines can access. Another pipeline can use a Run pipeline job node to call the pipeline that set the user status, and then get the user status using tasks.<node name>.results.output_parameters.<user status parameter name>.

If SetUserStatus is called in a child pipeline, migration creates a global user variable named user_status and selects the option Make user variable value available as a pipeline result. In the parent pipeline, it also replaces the expression that gets the status of the child pipeline, .$UserStatus, with tasks.results.output_parameters.user_status.

Constructed flows

Run DataStage job: The Environment Variables parameter value is overwritten if the DataStage flow includes the same environment. The List type is mapped to Enum. Path is mapped to the File type. For information see Configuring global objects for Orchestration Pipelines.
Run Bash script: Echo statements must use double quotes to access the value of a variable. For example, echo "variablename" will replace "variablename" with the value of the variable. echo 'variablename' will just echo the name of the variable.