Runtime column propagation (Java Integration stage in DataStage)

IBM® DataStage® allows you to define part of your schema and specify that if your job encounters extra columns that are not defined in the meta data when it actually runs, it will adopt these extra columns and propagate them through the rest of the job. This is known as runtime column propagation.

You can enable runtime column propagation for a project and set for individual links in the Output Page Columns tab. To enable runtime column propagation, select the Runtime column propagation check box .

Adding extra columns to output link

When runtime column propagation is enabled on a output link, your user code can add extra columns to this output link. The extra columns can be specified by Processor.getAdditionalOutputColumns() method, which is called for each output link whose runtime column propagation is enabled.

Following example shows how to add columns named "charCol" and "intCol" to a given output link.

public List<ColumnMetadata> getAdditionalOutputColumns
Link outputLink, List<Link> inputLinks, 
Properties stageProperties)
{
List<ColumnMetadata> addtionalColumns = new 
ArrayList<ColumnMetadata>();
ColumnMetadata charCol = new ColumnMetadataImpl
("charCol",ColumnMetadata.SQL_TYPE_CHAR);
addtionalColumns.add(charCol);
ColumnMetadata intCol = new ColumnMetadataImpl
("intCol",ColumnMetadata.SQL_TYPE_INTEGER);
addtionalColumns.add(intCol);
return addtionalColumns;
}

If you want to add columns that are in input link 0, but not in the output link, you can use the following helper method named the subtractColumnList() of the InputLink interface:


public List<ColumnMetadata> getAdditionalOutputColumns(Link outputLink, 
List<Link> inputLinks, Properties stageProperties)
{
return inputLinks.get(0).subtractColumnList(outputLink);^M
}

Alternatively, you can use JavaBean to specify column definitions on the output links at runtime. If your user code uses JavaBean on the output links whose runtime column propagation is enabled, and both the JavaBean class and the Column mapping property is empty, the Java™ Integration stage automatically creates columns from the bean properties in a given JavaBean class, and add them to an output link.

The following table summarizes the SQL type created by the Java Integration stage. You can overwrite SQL type of columns by manually adding the corresponding column definition on the output link.

Java type	SQL type
byte[]	Binary
boolean/java.lang.Boolean	Bit
short/java.lang.Short	TinyInt
int/java.lang.Integer	SmallInt
double/java.lang.Double	Double
float/java.lang.Float	Float
long/java.lang.Long	Integer
java.lang.String	VarChar
java.math.BigInteger	BigInt
java.math.BigDecimal	Decimal
java.sql.Date	Date
java.sql.Time	Time
com.ibm.is.cc.javastage.api.TimeMicroseconds.class	Time
java.sql.Timestamp	Timestamp

If the stage has a single input link and one or more output links, the Java Integration stage automatically adds all columns on the input link that are not present in the output link whose RCP is enabled.

Transferring the column data from input to output

Java Integration stage API provides the functionality to query column metadata dynamically at runtime, and access the data. You need to create code to read the data of the propagated columns on the input link, and write them to the output link for which the RCP is enabled.

Java Integration stage also provides the functionality to automatically transfer the column data from an input link to output link if the stage has a single input link and one or more output links. In this case, your user code is not required to transfer the data of the propagated columns on the input link. For more information, see the com.ibm.is.cc.javastage.api.ColumnTransferBehavior class in the Javadoc information for the Java Integration stage API.