Installing a Python module to set up model evaluation

Last updated: Nov 21, 2024

To automate the provisioning and configuration of the required services and see an evaluation application, including sample data, you can install a Python module.

About this module

The module provides an alternative way for technical users to see an instance of Watson OpenScale running without needing to provision and configure the services yourself, as described in the Getting started tutorial.
The Python module runs through the process of checking the services that you have and creating the ones that are necessary. After the module runs successfully, from the IBM Cloud dashboard you can start Watson OpenScale to see how it evaluates a model.

Watson OpenScale Python Client

The Python Client is a library that works directly with the model evaluation service. For development and automation purposes, you can use the Python client to directly configure the data mart database, access your machine learning engine, and select and evaluate deployments. For examples that use the Python client in this way, see the Watson OpenScale sample Notebooks.

Before you begin

Create an IBM Cloud API key and download it. You need to enter the API key in a later step.
Install any release of Python 3.

Python 3 includes the pip package management system.
Install the ibm-watson-openscale-cli package by running the following command:
```
pip install -U ibm-watson-openscale-cli
```
If more than one version of pip is installed on your system, you might need to run pip3 instead of pip, as in, pip3 install -U ibm-watson-openscale-cli.
If you have an existing watsonx.ai Runtime service instance, check the IBM Cloud dashboard to ensure that the service is managed by Cloud Identity and Access Management (IAM), not Cloud Foundry.

Important:
The module checks for an instance of watsonx.ai Runtime. If you have an instance, the module uses it. But if your instance is managed by Cloud Foundry, you must first migrate it to an IAM resource group before you run the module.

Running the module

Run the following command:

ibm-watson-openscale-cli -a <Your API key>

Viewing results

To view insights into the fairness and accuracy of the model, details of data that is evaluated, and explainability for an individual transaction, open the Insights dashboard.

To understand the scenario for the sample data, see Getting started.

View insights

From the Insights dashboard, click the Insights tab, which shows an overview of metrics for deployed models:

At a glance, the Insights page shows any issues with fairness and accuracy, as determined by the thresholds that are configured.
Each deployment is shown as a tile. If you are following any one of the tutorials, in your dashboard, you see a deployment that is called GermanCreditRiskModel. The following sample shows a dashboard with many deployed and evaluated models:

View evaluation data

From the Insights page, click the GermanCreditRiskModel tile to view details about the evaluated data.
Slide the marker across the chart to view a day and time period that show data and then click the View details link.
- For example, the following screen shows data for a specific date and time. The dates and times vary, depending on when you run the module.
- For information about interpreting the time series chart, see Getting insights.
To see details about AGE data evaluation, ensure that AGE is selected from the menu.
- Notice that in the following screen capture, no bias exists.
- For information about interpreting the chart of the data points at a specific hour, see Visualizing data for a specific hour.

View explainability

To understand the factors that contribute when bias is present for a given time period, from the visualization screen that is shown in the previous section, click View transactions.

Transaction IDs for the past hour are listed for those transactions that have bias. For the model used in this module, no bias exists for requests that are available. Therefore, no transactions are shown for the time period in the following screen capture.

Transaction list with no transactions

For more information, see Explaining transactions.

Related information

To learn about biases, see Fairness.
To learn about how well your model predicts outcomes, see Accuracy.
To learn about interpreting charts and data, see Getting insights.
To learn how underlying factors influence outcomes, see Monitoring explainability.

Parent topic: Watson OpenScale