distinctnode properties

Distinct node icon The Distinct node removes duplicate records, either by passing the first distinct record to the data flow or by discarding the first record and passing any duplicates to the data flow instead.

Example

node = stream.create("distinct", "My node")
node.setPropertyValue("mode", "Include")
node.setPropertyValue("fields", ["Age" "Sex"])
node.setPropertyValue("keys_pre_sorted", True)

Table 1. distinctnode properties
`distinctnode` properties	Data type	Property description
`mode`	`Include` `Discard`	You can include the first distinct record in the data stream, or discard the first distinct record and pass any duplicate records to the data stream instead.
`composite_value`	Structured slot	See example below.
`composite_values`	Structured slot	See example below.
`inc_record_count`	flag	Creates an extra field that specifies how many input records were aggregated to form each aggregate record.
`count_field`	string	Specifies the name of the record count field.
`default_ascending`	flag
`low_distinct_key_count`	flag	Specifies that you have only a small number of records and/or a small number of unique values of the key field(s).
`keys_pre_sorted`	flag	Specifies that all records with the same key values are grouped together in the input.
`disable_sql_generation`	flag
`grouping_fields`	array	Lists the field or fields used to determine whether records are identical.
`sort_keys`	array	Lists the fields used to determine how records are sorted within each group of duplicates, and whether they're sorted in ascending or descending order. You must specify a sort order if you've chosen to include or exclude the first record in each group, and if it matters to you which record is treated as the first.
`default_sort_order`	`Ascending` `Descending`	Specify whether, by default, records are sorted in ascending or descending order of the sort key values.
`existing_sort_keys`	array	Specify the existing sort order.

Example for `composite_value` property

The composite_value property has the following general form:

node.setKeyedPropertyValue("composite_value", FIELD, FILLOPTION)

FILLOPTION has the form [ FillType, Option1, Option2, ...].

Examples:

node.setKeyedPropertyValue("composite_value", "Age", ["First"])
node.setKeyedPropertyValue("composite_value", "Age", ["last"])
node.setKeyedPropertyValue("composite_value", "Age", ["Total"])
node.setKeyedPropertyValue("composite_value", "Age", ["Average"])
node.setKeyedPropertyValue("composite_value", "Age", ["Min"])
node.setKeyedPropertyValue("composite_value", "Age", ["Max"])
node.setKeyedPropertyValue("composite_value", "Date", ["Earliest"])
node.setKeyedPropertyValue("composite_value", "Date", ["Latest"])
node.setKeyedPropertyValue("composite_value", "Code", ["FirstAlpha"])
node.setKeyedPropertyValue("composite_value", "Code", ["LastAlpha"])

The custom options require more than one argument. These are added as a list, for example:

node.setKeyedPropertyValue("composite_value", "Name", ["MostFrequent", "FirstRecord"])
node.setKeyedPropertyValue("composite_value", "Date", ["LeastFrequent", "LastRecord"])
node.setKeyedPropertyValue("composite_value", "Pending", ["IncludesValue", "T", "F"]) 
node.setKeyedPropertyValue("composite_value", "Marital", ["FirstMatch", "Married", "Divorced", "Separated"])
node.setKeyedPropertyValue("composite_value", "Code", ["Concatenate"])
node.setKeyedPropertyValue("composite_value", "Code", ["Concatenate", "Space"])
node.setKeyedPropertyValue("composite_value", "Code", ["Concatenate", "Comma"])
node.setKeyedPropertyValue("composite_value", "Code", ["Concatenate", "UnderScore"])

Example for `composite_values` property

The composite_values property has the following general form:

node.setPropertyValue("composite_values", [
                        [FIELD1, [FILLOPTION1]],
                        [FIELD2, [FILLOPTION2]],
.
.
])

Example:

node.setPropertyValue("composite_values", [
    ["Age", ["First"]],
    ["Name", ["MostFrequent", "First"]],
    ["Pending", ["IncludesValue", "T"]],
    ["Marital", ["FirstMatch", "Married", "Divorced", "Separated"]],
    ["Code", ["Concatenate", "Comma"]]
])

Example for composite_value property

Example for composite_values property

Example for `composite_value` property

Example for `composite_values` property