Evaluation charts are similar to histograms or collection graphs. Evaluation charts show
how accurate models are in predicting particular outcomes. They work by sorting records based on the
predicted value and confidence of the prediction, splitting the records into groups of equal size
(quantiles), and then plotting the value of the criterion for each quantile, from highest to lowest.
Multiple models are shown as separate lines in the plot.
Outcomes are handled by defining a specific value or range of values as a "hit". Hits usually
indicate success of some sort (such as a sale to a customer) or an event of interest (such as a
specific medical diagnosis).
Flag
Output fields are straightforward; hits correspond to true values.
Nominal
For nominal output fields, the first value in the set defines a hit.
Continuous
For continuous output fields, hits equal values greater than the midpoint of the field's
range.
Evaluation charts can also be cumulative so that each point equals the value for the
corresponding quantile plus all higher quantiles. Cumulative charts usually convey the overall
performance of models better, whereas noncumulative charts often excel at indicating particular
problem areas for models.
Creating a simple Evaluation chart
Copy link to section
In the Chart Type section, click the Evaluation
icon.
The canvas updates to display an Evaluation chart template.
Set the Target field, Predict field and
Confidence field variables. The target field can be any instantiated flag or
nominal field with two or more values. The predict field defines the variable that is used as the
predicted value. The confidence field defines the variable that is used to establish the confidence
of the prediction.
Note: The Predict field variable type
must match the variable type that is selected for the Target
field.
Specify a custom condition used to indicate the User defined
hit. This option is useful for defining the outcome of interest rather than deducing it
from the type of target field and the order of values.
You must specify a CLEM
expression for a hit condition. For example, @TARGET = "YES" is a valid condition
that indicates a value of Yes for the target field is counted as a hit in the
evaluation. The specified condition is used for all target fields.
Click the Save visualization in the
project control. Select Create a new asset or Append
to existing asset. Provide a Visualization asset name, an optional description, and a
chart name.
Click Apply to save the visualization to
the project. The new visualization asset is now available on the Assets
tab.
Options
Copy link to section
Target field
Lists instantiated flag or nominal field variables with two or more values.
User defined hit
Specify a hit value. Hits indicate events of interest (for example, a specific medical
diagnosis).
Predict field
Lists variables that can be used as the predicted value.
Confidence field
Lists variables that can establish the confidence of the prediction.
Cumulative plot
Create a cumulative chart when enabled. Values in cumulative charts are
plotted for each quantile plus all higher quantiles.
Display mode
The settings control which charts display in preview mode and in the output.
Single mode
When selected, the Model Classification Tuning chart is in the only chart that displays in
preview mode and in the output.
Classical mode
When selected, the Model Classification Tuning, Cutoff, Matrix Bar, ROC,
Gains, ROI, and Profit charts display in preview mode and in the output.
Full mode
When selected, the Model Classification Tuning, Cutoff, Matrix Bar, ROC, Gains, ROI, Profit,
GINI, Lift, and Response charts display in preview mode and in the output.
Evaluation charts
Cutoff
The cutoff chart shows the predicted versus actual values for selected variables for a specified
cutoff value.
Matrix Bar
Matrix Bar charts are a good way to determine whether linear correlations exist between multiple
variables.
ROC
ROC (Receiver Operating Characteristic) evaluates the performance of classification schemes
where subjects are classified for one variable with two categories.
Gains
Gains are defined as the proportion of total hits that occurs in each quantile. Gains are
computed as (number of hits in quantile / total number of hits) × 100%.
ROI
ROI (return on investment) is similar to profit in that it involves defining revenues and costs.
ROI compares profits to costs for the quantile. ROI is computed as (profits for quantile /
costs for quantile) × 100%.
Profit
Profit equals the revenue for each record minus the cost for the record. Profits for a quantile
are the sum of profits for all records in the quantile. Revenues are assumed to apply only to hits,
but costs apply to all records. Profits and costs can be fixed or can be defined by fields in the
data. Profits are computed as (sum of revenue for records in quantile − sum of costs for records in
quantile).
Kolmogorov-Smirnov
Compares the observed cumulative distribution function for a variable with a
specified theoretical distribution, which can be normal, uniform, exponential, or Poisson.
GINI
GINI measures statistical dispersion and is intended to represent the income or wealth
distribution. It is the most commonly used measurement of inequality.
Lift
Lift compares the percentage of records in each quantile that are hits with the overall
percentage of hits in the training data. It is computed as (hits in quantile / records in
quantile) / (total hits / total records).
Response
Response is the percentage of records in the quantile that are hits. Response is computed as
(hits in quantile / records in quantile) × 100%.
Evaluation chart settings
The following settings apply only to profit and ROI charts.
Costs
Specify the fixed cost associated with each record.
Revenue
Specify the fixed revenue associated with each record that represents a hit.
Weight
If the records in your data represent more than one unit, you can use frequency weights to
adjust the results. Specify the fixed weight associated with each record.
About cookies on this siteOur websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising.For more information, please review your cookie preferences options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.