Troubleshoot watsonx.ai Runtime

Running an AutoAI time series experiment with anomaly prediction fails

The feature to predict anomalies in the results of a time series experiment is no longer supported. Trying to run an existing experiment results in errors for missing runtime libraries. For example, you might see this error:

The selected environment seems to be invalid: Could not retrieve environment. CAMS error: Missing or invalid asset id

The behavior is expected as the runtimes for anomaly prediction are not supported. There is no workaround for this problem.

AutoAI inference notebook for a RAG experiment exceeds model limits

Sometimes when running an inference notebook generated for an AutoAI RAG experiment, you might get this error:

MissingValue: No "model_limits" provided. Reason: Model <model-nam> limits cannot be found in the model details.

The error indicates that the token limits for inferencing the foundation model used for the experiment are missing. To resolve the problem, find the function default_inference_function and replace get_max_input_tokens with the max tokens for the model. For example:

model = ModelInference(api_client=client, **params['model"])
# model_max_input_tokens = get+max_input_tokens(model=model, params=params)
model_max_input_tokens = 4096

You can find the max token value for the model in the table of Supported foundation models available with watsonx.ai.

Training an AutoAI experiment fails with service ID credentials

If you are training an AutoAI experiment using the API key for the serviceID, training might fail with this error:

User specified in query parameters does not match user from token.

One way to resolve this issue is to run the experiment with your user credentials. If you want to run the experiment with credentials for the service, follow these steps to update the roles and policies for the service ID.

1. Open your serviceID on IBM Cloud.
2. Create a new serviceID or update the existing ID with the following access policy:
   • All IAM Account Management services with the roles API key reviewer,User API key creator, Viewer, Operator, and Editor. Ideally it is best if they create a new apikey for this ServiceId.
3. The updated policy will look as follows:
4. Run the training again with the credentials for the updated serviceID.

Prediction request for AutoAI time-series model can time out with too many new observations

A prediction request can time out for a deployed AutoAI time-series model if there are too many new observations passed. To resolve the problem, do one of the following:

• Reduce the number of new observations.
• Extend the training data used for the experiment by adding new observations. Then, re-run the AutoAI time-series experiment with the updated training data.

Insufficient class members in training data for AutoAI experiment

Training data for an AutoAI experiment must have at least 4 members for each class. If your training data has an insufficient number of members in a class, you will encounter this error:

ERROR: ingesting data Message id: AC10011E. Message: Each class must have at least 4 members. The following classes have too few members: ['T'].

To resolve the problem, update the training data to remove the class or add more members.

Batch deployments that use large data volumes as input might fail

If you are scoring a batch job that uses large volumes of data as the input source, the job might fail becase of internal timeout settings. A symptom of this problem might be an error message similar to the following example:

Incorrect input data: Flight returned internal error, with message: CDICO9999E: Internal error occurred: Snowflake sQL logged error: JDBC driver internal error: Timeout waiting for the download of #chunk49(Total chunks: 186) retry=0.

If the timeout occurs when you score your batch deployment, you must configure the data source query level timeout limitation to handle long-running jobs.

Query-level timeout information for data sources is as follows:

To avoid your batch deployments from failing, partition your data set or decrease its size.

Security for file uploads

Files you upload through the watsonx.ai Studio or watsonx.ai Runtime UI are not validated or scanned for potentially malicious content. It is recommended that you run security software, such as an anti-virus application, on all files before uploading to ensure the security of your content.

Deployments with constricted software specifications fail after an upgrade

If you upgrade to a more recent version of IBM Cloud Pak for Data and deploy an R Shiny application asset that was created by using constricted software specifications in FIPS mode, your deployment fails.

For example, deployments that use shiny-r3.6 and shiny-r4.2 software specifications fail after you upgrade from IBM Cloud Pak for Data version 4.7.0 to 4.8.4 or later. You might receive the error message Error 502 - Bad Gateway.

To prevent your deployment from failing, update the constricted specification for your deployed asset to use the latest software specification. For more information, see Managing outdated software specifications or frameworks. You can also delete your application deployment if you no longer need it.

Creating a job for an SPSS Modeler flow in a deployment space fails

During the process of configuring a batch job for your SPSS Modeler flow in a deployment space, the automatic mapping of data assets with their respective connection might fail.

To fix the error with the automatic mapping of data assets and connections, follow these steps:

1. Click Create to save your progress and exit from the New job configuration dialog box.

2. In your deployment space, click the Jobs tab and select your SPSS Modeler flow job to review the details of your job.

3. In the job details page, click the Edit icon to manually update the mapping of your data assets and connections.

4. After updating the mapping of data assets and connection, you can resume with the process of configuring settings your job in the New job dialog box. For more information, see Creating deployment jobs for SPSS Modeler flows

   

Parent topic: Troubleshooting