IBM Cloud Object Storage connection

Last updated: Nov 27, 2024

To access your data in IBM Cloud Object Storage (COS), create a connection asset for it.

IBM Cloud Object Storage on IBM Cloud provides unstructured data storage for cloud applications. Cloud Object Storage offers S3 API and application binding with regional and cross-regional resiliency.

Important:

An IBM Cloud Object Storage connection has a different purpose from the IBM Cloud Object Storage instance that you associate with a project, deployment space, or catalog. You create an IBM Cloud Object Storage connection when you want to create a connected data asset that connects to data stored in IBM Cloud Object Storage. You associate an IBM Cloud Object Storage instance when you create projects, deployment spaces, or catalogs to store files for assets, such as uploaded data files or notebook files.

Create a connection to IBM Cloud Object Storage

If you have set up an integrated cloud service, select the service instance to automatically fill in the fields in the connection form. Confirm that all the fields are complete.

To create the connection asset, you need these connection details:

Bucket name. (Optional. If you do not enter the bucket name, then the credentials must have permission to list all the buckets.)
Login URL. To find the Login URL:

Go to the Cloud Object Storage Resource list at https://cloud.ibm.com/resources.
Expand the Storage resource.
Click the Cloud Object Storage service. From the menu, select Endpoints.
Optional: Use the Select resiliency and Select location menus to filter the choices.
Copy the value of the public endpoint that is in the same region as the bucket that you want to use.

SSL certificate: (Optional). A self-signed certificate that was created by a tool such as OpenSSL.

Credentials

Use one of the following combination of values for authentication:

Service credentials
Resource instance ID and API key
Resource instance ID, API key, Access key, and Secret key (In this combination, the Resource instance ID and API key are used for authentication. The Access key and Secret key are stored.)
Access key and Secret key

To find the value for Service credentials:

Go to the Cloud Object Storage Resource list at https://cloud.ibm.com/resources.
Expand the Storage resource.
Click the Cloud Object Storage service, and then click the Service credentials tab.
Expand the Key name that you want to use.
Copy the entire JSON file. Include the opening and closing braces { } symbols.

To find the values for the API key, Access key, Secret key, and the Resource instance ID:

Go to the Cloud Object Storage Resource list at https://cloud.ibm.com/resources.
Expand the Storage resource.
Click the Cloud Object Storage service, and then click the Service credentials tab.
Expand the Key name that you want to use. Copy the values without the quotation marks:

API key: apikey
Access key: access_key_id
Secret key: secret_access_key
Resource instance ID: resource_instance_id

Choose the method for creating a connection based on where you are in the platform

In a project: Click Assets > New asset > Connect to a data source. See Adding a connection to a project.
In a catalog: Click Add to catalog > Connection. See Adding a connection asset to a catalog.
In a deployment space: Click Import assets > Data access > Connection. See Adding data assets to a deployment space.
In the Platform assets catalog: Click New connection. See Adding platform connections.

Next step: Add data assets from the connection

Where you can use this connection

You can use IBM Cloud Object Storage connections in the following workspaces and tools:

Projects

AutoAI (watsonx.ai Runtime)
Data Refinery (watsonx.ai Studio or IBM Knowledge Catalog)
DataStage (DataStage service). See Connecting to a data source in DataStage.
Decision Optimization (watsonx.ai Studio and watsonx.ai Runtime)
Metadata enrichment (IBM Knowledge Catalog)
Metadata import (IBM Knowledge Catalog)
SPSS Modeler (watsonx.ai Studio)

Catalogs

Platform assets catalog
Other catalogs (IBM Knowledge Catalog). To enable the preview of text and image assets within a catalog, the credentials for the connection must include the access key and the secret key in addition to the API key and the resource instance ID.

Data Virtualization service: You can connect to this data source from Data Virtualization. This connection requires special consideration in Data Virtualization. See Connecting to Cloud Object Storage in Data Virtualization.

Connecting to the Cloud Object Storage service with the S3 API

To connect to Cloud Object Storage with the S3 API, you need the Login URL, an Access key and a Secret key.

The API key is a token that is used to call the Watson IoT Platform HTTP APIs. Users are assigned roles and they can generate an API key that they can use to authorize calls to API endpoints. For more information, see the IBM Cloud Object Storage S3 API documentation.

IBM Cloud Object Storage setup

Getting started with IBM Cloud Object Storage

Restrictions

The following restrictions apply:

For DataStage, you must create the Cloud Object Storage credentials with the Hash-based Message Authentication Code (HMAC) option. See Using HMAC credentials.

Supported file types

The IBM Cloud Object Storage connection supports these file types: Avro, CSV, Delimited text, Excel, JSON, ORC, Parquet, SAS, SAV, SHP, and XML.

Table formats

The IBM Cloud Object Storage connection supports these Data Lake table formats: Delta Lake and Iceberg.

Learn more

Controlling access to COS buckets

Parent topic: Supported connections