Managing storage space in Cloud Object Storage

Last updated: Nov 27, 2024

You can monitor and manage the storage buckets for workspaces from the Cloud Object Storage service instance page in IBM Cloud.

Cloud Object Storage provides a storage bucket for each project, deployment space, and catalog workspace. A Cloud Object Storage bucket is automatically created when you create the workspace. Files for assets that you explicitly add or create in a workspace are stored in the bucket. Various tools also automatically store system files such as logs in the workspace bucket. You are billed monthly for any storage that exceeds the amount for the Free Tier of the Standard plan. For more information about Cloud Object Storage plans, see IBM Cloud Object Storage service plans.

Required permissions

To view or manage Cloud Object Storage in IBM Cloud, you must have one of the following roles in the IBM Cloud account:

The Owner or Administrator role for the IBM Cloud account.

The Manager or Writer IAM role for the IBM Cloud Object Storage service.

Identifying the Cloud Object Storage instance and bucket for a workspace

If you are a collaborator in a workspace, you can identify the Cloud Object Storage instance and the storage bucket for that workspace.

For a project or a deployment space, the bucket information is on the Manage page, in the Storage section. If you click the bucket name, you go to IBM Cloud and see the Buckets list for the Cloud Object Storage instance that is associated with the project or deployment space. The Cloud Object Storage service instance name is shown at the top of the page.

For a project or a deployment space, the Cloud Object Storage instance name and the bucket information are on the Manage page, in the Storage section.

For a catalog, the bucket information is on the Settings page, in the Cloud Object Storage section.

Managing Cloud Object Storage in IBM Cloud

You can administer Cloud Object Storage from the Resource list > Storage page on the IBM Cloud dashboard. For example, you can upload and download assets, manage buckets, and configure credentials and other security settings for the Cloud Object Storage instance.

To manage the Cloud Object Storage instance for Cloud Pak for Data as a Service, from the navigation menu, select Services > Service instances and then click the name of the Cloud Object Storage service instance. The instance page opens in IBM Cloud. You can click a bucket name to view its files, download files, or perform other task.

Watch this video to see how to manage an object storage instance.

This video provides a visual method to learn the concepts and tasks in this documentation.

Transcript

Synchronize transcript with video

Video transcript
Time	Transcript
00:00	This video shows how to manage an IBM Cloud Object Storage instance.
00:06	When you create a project, an IBM Cloud Object Storage instance is associated with the project.
00:15	On the Manage tab, you'll see the associated object storage instance and have the option to manage it in IBM Cloud.
00:24	IBM Cloud Object Storage uses buckets to organize your data.
00:30	You can see that this instance contains a bucket with the "jupyternotebooks" prefix, which was created when the "Jupyter Notebooks" project was created.
00:41	If you open that bucket, you'll see all of the files that you added to that project.
00:47	From here, you can download an object or delete it from the bucket.
00:53	You can also view the object SQL URL to access that object from your application.
01:00	You can add objects to the bucket from here.
01:03	Just browse to select the file and wait for it to upload to storage.
01:10	And then that file will be available in the Files slide-out panel in the project.
01:16	Let's create a bucket.
01:20	You can create a Standard or Archive bucket, based on predefined settings, or create a custom bucket.
01:28	Provide a bucket name, which must be unique across the IBM Cloud Object Storage system.
01:35	Select a resiliency.
01:38	Cross Region provides higher availability and durability and Regional provides higher performance.
01:45	The Single Site option will only distribute data across devices within a single site.
01:52	Then select the location based on workload proximity.
01:57	Next, select a storage class, which defines the cost of storing data based on frequency of access.
02:05	Smart Tier provides automatic cost optimization for your storage.
02:11	Standard indicates frequent access.
02:14	Vault is for less frequent access.
02:18	And Cold Vault is for rare access.
02:21	There are other, optional settings to add rules, keys, and services.
02:27	Refer to the documentation for more details on these options.
02:32	When you're ready, create the bucket.
02:35	And, from here, you could add files to that bucket.
02:40	On the Access policies panel, you can manage access to buckets using IAM policies - that's Identity and Access Management.
02:50	On the Configuration panel, you'll find information about Key Protect encryption keys, as well as the bucket instance CRN and endpoints to access the data in the buckets from your application.
03:01	You can also find some of the same information on the Endpoints panel.
03:06	On the Service credentials panel, you'll find the API and access keys to authenticate with your instance from your application.
03:15	You can also connect the object storage to a Cloud Foundry application, check usage details, and view your plan details.
03:26	Find more videos in the Cloud Pak for Data as a Service documentation.

Adding files to a bucket from IBM Cloud

You can add files to a bucket by working with your Cloud Object Storage bucket in IBM Cloud. However, those files are not project assets until you convert them to assets. See Convert files in project storage to assets.

Deleting files from a bucket

The recommended practice for managing Cloud Object Storage resources is to delete assets that are no longer needed from the workspace in Cloud Pak for Data as a Service. Deleting an asset from the workspace automatically deletes all the associated user-created and system-generated files from the storage bucket. For example, deleting an AutoAI experiment removes the binary files for each model-candidate pipeline, the auto-generated notebooks, and the metadata files from the Cloud Object Storage bucket.

Important:

Contact IBM Support for assistance with deleting items from Cloud Object Storage buckets in IBM Cloud to free up space. Deleting items from Cloud Object Storage buckets instead of from workspaces can adversely affect the integrity of projects, catalogs, or spaces and make them inoperative.

You can delete the asset from within a workspace in the UI or from the command line. For example, you can delete an AutoAI asset and all its supporting files by using the watsonx.ai Runtime REST API or the The ibm-watson-machine-learning Python library.

To delete an AutoAI asset by using the API, run the following command after replacing {id} with the asset ID:

DELETE /ml/api/v4/trainings/{id}?hard_delete=true

To delete an AutoAI asset by using the ibm-watson-machine-learning Python SDK, run the following command after replacing {id} with the asset ID:

client.trainings.cancel(training_uid={id}, hard_delete=True)

Learn more

Parent topic: Creating a project