Frequently asked questions

To run Watson OpenScale on my own servers, how much computer processing power is required?

There are specific guidelines for hardware configuration for three-node and six-node configurations. Your IBM Technical Sales team can also help you with sizing your specific configuration. Because Watson OpenScale runs as an add-on to IBM Cloud Pak for Data, you need to consider the requirements for both software products.

Why does Watson OpenScale need access to training data?

You must either provide Watson OpenScale access to training data that is stored in Db2 or IBM Cloud Object Storage, or you must run a Notebook to access the training data.

Watson OpenScale needs access to your training data for the following reasons:

• To generate Local Interpretable Model-Agnostic Explanations (LIME) and Contrastive explanations: To create explanations, access to statistics, such as median value, standard deviation, and distinct values from the training data is required.
• To display training data statistics: To populate the bias details page, Watson OpenScale must have training data from which to generate statistics.
• To build a drift detection model: The Drift monitor uses training data to create and calibrate drift detection.

In the Notebook-based approach, you are expected to upload the statistics and other information when you configure a deployment in Watson OpenScale. Watson OpenScale no longer has access to the training data outside of the Notebook, which is run in your environment. It has access only to the information uploaded during the configuration.

Is there a command-line tool to use?

Yes! There is a ModelOps CLI tool, whose official name is the Watson OpenScale CLI model operations tool. Use it to run tasks related to the lifecycle management of machine learning models.

What version of Python can I use with Watson OpenScale?

Because Watson OpenScale is independent of your model-creation process, it supports whatever Python versions your machine learning provider supports. The Watson OpenScale Python client is a Python library that works directly with the Watson OpenScale service on IBM Cloud. For the most up-to-date version information, see the Requirements section. You can use the Python client, instead of the Watson OpenScale client UI, to directly configure a logging database, bind your machine learning engine, and select and monitor deployments. For examples of using the Python client in this way, see the Watson OpenScale sample Notebooks.

Which browsers can I use to run Watson OpenScale?

The Watson OpenScale service tooling requires the same level of browser software as is required by IBM Cloud. See the IBM Cloud Prerequisites topic for details.

Configuring a model requires information about the location of the training data and the options are Cloud Object Storage and Db2. If the data is in Netezza, can Watson OpenScale use Netezza?

Use this Watson OpenScale Notebook to read the data from Netezza and generate the training statistics and also the drift detection model.

Why doesn't Watson OpenScale see the updates that were made to the model?

Watson OpenScale works on a deployment of a model, not on the model itself. You must create a new deployment and then configure this new deployment as a new subscription in Watson OpenScale. With this arrangement, you are able to compare the two versions of the model.

How is model bias mitigated by using Watson OpenScale?

The debiasing capability in Watson OpenScale is enterprise grade. It is robust, scalable and can handle a wide variety of models. Debiasing in Watson OpenScale consists of a two-step process.

1. Learning Phase: Learning customer model behavior to understand when it acts in a biased manner.
2. Application Phase: Identifying whether the customer’s model acts in a biased manner on a specific data point and, if needed, fixing the bias. For more information, see Reviewing debiased transactions.

Is it possible to check for model bias on sensitive attributes, such as race and sex, even when the model is not trained on them?

Yes. Recently, Watson OpenScale delivered a ground-breaking feature called “Indirect Bias detection.” Use it to detect whether the model is exhibiting bias indirectly for sensitive attributes, even though the model is not trained on these attributes. For more information, see Reviewing debiased transactions.

Can I configure model fairness through an API?

Yes, it is possible with the Watson OpenScale SDK. For more information, see IBM Watson OpenScale Python SDK documentation.

What are the various kinds of risks associated in using a machine learning model?

Multiple kinds of risks that are associated with machine learning models, such as any change in input data (also known as drift) can cause the model to make inaccurate decisions, impacting business predictions. Training data can be cleaned to be free from bias but runtime data might induce biased behavior of the model.

Traditional statistical models are simpler to interpret and explain, but unable to explain the outcome of the machine learning model can pose a serious threat to the usage of the model.

For more information, see Manage model risk.

Does Watson OpenScale detect drift in accuracy and drift in data?

Watson OpenScale detects both drift in accuracy and drift in data:

• Drift in accuracy estimates the drop in accuracy of the model at run time. Model accuracy drops when there is an increase in transactions that are similar to those that the model did not evaluate correctly in the training data. This type of drift is calculated for structured binary and multi-class classification models only.
• Drift in data estimates the drop in consistency of the data at runtime as compared to the characteristics of the data at training time.

What are the types of explanations shown in Watson OpenScale?

Watson OpenScale provides Local explanation based on LIME, contrastive explanation, and SHAP explanations. For more information, see Viewing explainability.

What is what-if analysis in Watson OpenScale?

The explanations UI also provides the ability to test what-if scenarios. For example, the user can change the feature values of the input datapoint and check its impact on the model prediction and probability.

Which models support LIME explanations?

The following types of models support LIME explanations in Watson OpenScale:

• Structured regression and classification models
• Unstructured text and image classification models

Which models support contrastive explanations and what-if analysis?

Contrastive explanations and what-if analyses are supported for models that use structured data and classification problems only.

What are controllable features in Watson OpenScale explainability configuration?

Using controllable features some features of the input data point can be locked, so that they do not change when the contrastive explanation is generated and also they cannot be changed in what if analysis. The features that should not be changed should be set as noncontrollable or NO in the explainability configuration.

While configuring the machine learning providers in Watson OpenScale, what is the difference between pre-production and production subscriptions?

Before putting a model into production, a model validator can configure and validate the model by using a pre-production service provider. Watson OpenScale can designate a machine learning provider as part of pre-production processing, where you perform risk evaluations. After the model meets quality standards, then you can send the model to production. You can use different machine learning providers for pre-production and production, or you can use different instances of the same machine learning provider to keep pre-production and production environments segmented.

Must I keep monitoring the Watson OpenScale dashboard to make sure that my models behave as expected?

No, you can set up email alerts for your production model deployments in Watson OpenScale. Email alerts are sent whenever a risk evaluation test fails, and then you can come and check the issues and address them.

In a pre-production environment that uses Watson OpenScale after the model is evaluated for risk and approved for usage, do I have to reconfigure all the monitors again in production environment?

No, Watson OpenScale provides a way to copy the configuration of pre-production subscription to production subscription. For more information, see Manage model risk.

In Watson OpenScale, can I compare my model deployments in pre-production with a benchmark model to see how good or bad it is?

Yes, Watson OpenScale provides you with the option to compare two model deployments or subscriptions. You can see a side-by-side comparison of the behavior of the two models on each of the monitors configured. To compare go to the model summary page on Watson OpenScale dashboard and select Actions -> Compare. For more information, see Reviewing evaluation results.

In Watson OpenScale, what data is used for Quality metrics computation?

Quality metrics are calculated that use manually labeled feedback data and monitored deployment responses for this data.

In Watson OpenScale, why are some of the configuration tabs disabled?

Some conditions enable particular tabs. You can see the reason why that tab is not enabled, by hovering your mouse over the circle icon on the tab.

Why should one be concerned about model accuracy drift or data drift?

A drop in either model accuracy or data consistency leads to a negative impact on the business outcomes that are associated with the model and must be addressed by retraining the model.

What are the different kinds of drift that IBM Watson OpenScale detects?

Watson OpenScale detects both drift in model accuracy and drift in data.

How is drop in accuracy that is, model accuracy drift calculated in Watson OpenScale?

Watson OpenScale learns the behavior of the model by creating a proxy model, also known as a drift detection model. It looks at the training data and how the model is making predictions on the training data.

For more information, see Drift detection.

How is the drop in data consistency calculated in IBM Watson OpenScale?

IBM Watson OpenScale learns single and two-column constraints or boundaries on the training data at the time of configuration. It then analyzes all payload transactions to determine which transactions are causing drop in data consistency. For more information, see Drift metrics.

Can Watson OpenScale detect drift in my model that is trained on text corpus?

Watson OpenScale cannot detect drift in text-based models as of now.

Can Watson OpenScale detect drift in my model that is trained on image data?

Watson OpenScale cannot detect drift in image-based models as of now.

Can Watson OpenScale detect drift in my Python function that is deployed on IBM watsonx.ai Runtime?

Watson OpenScale can detect drift for Python functions that are trained on structured datasets. The python functions that were trained on text or image datasets are not supported. The scoring output for the python functions must include prediction columns and probability columns that contain all of the class probabilities for classification problems.

Parent topic: Evaluating AI models with Watson OpenScale