w.volumes: Volumes

class databricks.sdk.service.catalog.VolumesAPI

Volumes are a Unity Catalog (UC) capability for accessing, storing, governing, organizing, and processing files. Use cases include running machine learning on unstructured data such as image, audio, video, or PDF files, organizing data sets during the data exploration stages in data science, working with libraries that require access to the local file system on cluster machines, storing library and config files of arbitrary formats such as .whl or .txt centrally and providing secure access to those files across workspaces, or transforming and querying non-tabular data files in ETL.

create(catalog_name: str, schema_name: str, name: str, volume_type: VolumeType [, comment: Optional[str], storage_location: Optional[str]]) → VolumeInfo

Usage:

import os
import time

from databricks.sdk import WorkspaceClient
from databricks.sdk.service import catalog

w = WorkspaceClient()

storage_credential = w.storage_credentials.create(
    name=f"sdk-{time.time_ns()}",
    aws_iam_role=catalog.AwsIamRoleRequest(role_arn=os.environ["TEST_METASTORE_DATA_ACCESS_ARN"]),
    comment="created via SDK",
)

external_location = w.external_locations.create(
    name=f"sdk-{time.time_ns()}",
    credential_name=storage_credential.name,
    comment="created via SDK",
    url="s3://" + os.environ["TEST_BUCKET"] + "/" + f"sdk-{time.time_ns()}",
)

created_catalog = w.catalogs.create(name=f"sdk-{time.time_ns()}")

created_schema = w.schemas.create(name=f"sdk-{time.time_ns()}", catalog_name=created_catalog.name)

created_volume = w.volumes.create(
    catalog_name=created_catalog.name,
    schema_name=created_schema.name,
    name=f"sdk-{time.time_ns()}",
    storage_location=external_location.url,
    volume_type=catalog.VolumeType.EXTERNAL,
)

# cleanup
w.storage_credentials.delete(name=storage_credential.name)
w.external_locations.delete(name=external_location.name)
w.schemas.delete(full_name=created_schema.full_name)
w.catalogs.delete(name=created_catalog.name, force=True)
w.volumes.delete(name=created_volume.full_name)

Creates a new volume.

The user could create either an external volume or a managed volume. An external volume will be created in the specified external location, while a managed volume will be located in the default location which is specified by the parent schema, or the parent catalog, or the Metastore.

For the volume creation to succeed, the user must satisfy the following conditions:

  • The caller must be a metastore admin, or be the owner of the parent catalog and schema, or have the USE_CATALOG privilege on the parent catalog and the USE_SCHEMA privilege on the parent schema.

  • The caller must have CREATE VOLUME privilege on the parent schema.

For an external volume, the following conditions must also be satisfied:

  • The caller must have CREATE EXTERNAL VOLUME privilege on the external location.

  • There are no other tables, nor volumes existing in the specified storage location.

  • The specified storage location is not under the location of other tables, nor volumes, or catalogs or schemas.

Parameters:
  • catalog_name – str The name of the catalog where the schema and the volume are

  • schema_name – str The name of the schema where the volume is

  • name – str The name of the volume

  • volume_type – VolumeType The type of the volume. An external volume is located in the specified external location. A managed volume is located in the default location which is specified by the parent schema, or the parent catalog, or the Metastore. Learn more

  • comment – str (optional) The comment attached to the volume

  • storage_location – str (optional) The cloud storage URL where the files of the volume are stored. For an external volume, it must be set on creation to a path within an existing external location. For a managed volume, it must be omitted on creation; the volume is placed under the managed storage location of the parent schema, catalog, or metastore, and the assigned path is returned in this field.

Returns:

VolumeInfo

delete(name: str)

Deletes a volume from the specified parent catalog and schema.

The caller must be a metastore admin or an owner of the volume. For the latter case, the caller must also be the owner or have the USE_CATALOG privilege on the parent catalog and the USE_SCHEMA privilege on the parent schema.

Parameters:

name – str The three-level (fully qualified) name of the volume

list(catalog_name: str, schema_name: str [, include_browse: Optional[bool], max_results: Optional[int], page_token: Optional[str]]) → Iterator[VolumeInfo]

Usage:

import time

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

created_catalog = w.catalogs.create(name=f"sdk-{time.time_ns()}")

created_schema = w.schemas.create(name=f"sdk-{time.time_ns()}", catalog_name=created_catalog.name)

all_volumes = w.volumes.list(catalog_name=created_catalog.name, schema_name=created_schema.name)

# cleanup
w.schemas.delete(full_name=created_schema.full_name)
w.catalogs.delete(name=created_catalog.name, force=True)

Gets an array of volumes for the current metastore under the parent catalog and schema.

The returned volumes are filtered based on the privileges of the calling user. For example, the metastore admin is able to list all the volumes. A regular user needs to be the owner or have the READ VOLUME privilege on the volume to receive the volumes in the response. For the latter case, the caller must also be the owner or have the USE_CATALOG privilege on the parent catalog and the USE_SCHEMA privilege on the parent schema.

There is no guarantee of a specific ordering of the elements in the array.

PAGINATION BEHAVIOR: The API is by default paginated, a page may contain zero results while still providing a next_page_token. Clients must continue reading pages until next_page_token is absent, which is the only indication that the end of results has been reached.

Parameters:
  • catalog_name – str The name of the catalog that contains the schema.

  • schema_name – str The name of the schema whose volumes are listed.

  • include_browse – bool (optional) Whether to include volumes for which the principal can only access selective metadata through the BROWSE privilege.

  • max_results –

    int (optional) Maximum number of volumes to return (page length).

    If not set, the page length is set to a server configured value (10000, as of 1/29/2024).

    • when set to a value greater than 0, the page length is the minimum of this value and a server configured value (10000, as of 1/29/2024);

    • when set to 0, the page length is set to a server configured value (10000, as of 1/29/2024) (recommended);

    • when set to a value less than 0, an invalid parameter error is returned;

    Note: this parameter controls only the maximum number of volumes to return. The actual number of volumes returned in a page may be smaller than this value, including 0, even if there are more pages.

  • page_token – str (optional) Opaque token returned by a previous request. It must be included in the request to retrieve the next page of results (pagination).

Returns:

Iterator over VolumeInfo

read(name: str [, include_browse: Optional[bool]]) → VolumeInfo

Usage:

import os
import time

from databricks.sdk import WorkspaceClient
from databricks.sdk.service import catalog

w = WorkspaceClient()

storage_credential = w.storage_credentials.create(
    name=f"sdk-{time.time_ns()}",
    aws_iam_role=catalog.AwsIamRoleRequest(role_arn=os.environ["TEST_METASTORE_DATA_ACCESS_ARN"]),
    comment="created via SDK",
)

external_location = w.external_locations.create(
    name=f"sdk-{time.time_ns()}",
    credential_name=storage_credential.name,
    comment="created via SDK",
    url="s3://" + os.environ["TEST_BUCKET"] + "/" + f"sdk-{time.time_ns()}",
)

created_catalog = w.catalogs.create(name=f"sdk-{time.time_ns()}")

created_schema = w.schemas.create(name=f"sdk-{time.time_ns()}", catalog_name=created_catalog.name)

created_volume = w.volumes.create(
    catalog_name=created_catalog.name,
    schema_name=created_schema.name,
    name=f"sdk-{time.time_ns()}",
    storage_location=external_location.url,
    volume_type=catalog.VolumeType.EXTERNAL,
)

loaded_volume = w.volumes.read(name=created_volume.full_name)

# cleanup
w.storage_credentials.delete(name=storage_credential.name)
w.external_locations.delete(name=external_location.name)
w.schemas.delete(full_name=created_schema.full_name)
w.catalogs.delete(name=created_catalog.name, force=True)
w.volumes.delete(name=created_volume.full_name)

Gets a volume from the metastore for a specific catalog and schema.

The caller must be a metastore admin or an owner of (or have the READ VOLUME privilege on) the volume. For the latter case, the caller must also be the owner or have the USE_CATALOG privilege on the parent catalog and the USE_SCHEMA privilege on the parent schema.

Parameters:
  • name – str The fully qualified name of the volume.

  • include_browse – bool (optional) Whether to include volumes for which the principal can only access selective metadata through the BROWSE privilege.

Returns:

VolumeInfo

update(name: str [, comment: Optional[str], new_name: Optional[str], owner: Optional[str]]) → VolumeInfo

Usage:

import os
import time

from databricks.sdk import WorkspaceClient
from databricks.sdk.service import catalog

w = WorkspaceClient()

storage_credential = w.storage_credentials.create(
    name=f"sdk-{time.time_ns()}",
    aws_iam_role=catalog.AwsIamRoleRequest(role_arn=os.environ["TEST_METASTORE_DATA_ACCESS_ARN"]),
    comment="created via SDK",
)

external_location = w.external_locations.create(
    name=f"sdk-{time.time_ns()}",
    credential_name=storage_credential.name,
    comment="created via SDK",
    url="s3://" + os.environ["TEST_BUCKET"] + "/" + f"sdk-{time.time_ns()}",
)

created_catalog = w.catalogs.create(name=f"sdk-{time.time_ns()}")

created_schema = w.schemas.create(name=f"sdk-{time.time_ns()}", catalog_name=created_catalog.name)

created_volume = w.volumes.create(
    catalog_name=created_catalog.name,
    schema_name=created_schema.name,
    name=f"sdk-{time.time_ns()}",
    storage_location=external_location.url,
    volume_type=catalog.VolumeType.EXTERNAL,
)

loaded_volume = w.volumes.read(name=created_volume.full_name)

_ = w.volumes.update(name=loaded_volume.full_name, comment="Updated volume comment")

# cleanup
w.storage_credentials.delete(name=storage_credential.name)
w.external_locations.delete(name=external_location.name)
w.schemas.delete(full_name=created_schema.full_name)
w.catalogs.delete(name=created_catalog.name, force=True)
w.volumes.delete(name=created_volume.full_name)

Updates the specified volume under the specified parent catalog and schema.

The caller must be a metastore admin or an owner of the volume. For the latter case, the caller must also be the owner or have the USE_CATALOG privilege on the parent catalog and the USE_SCHEMA privilege on the parent schema.

Currently only the name, the owner, or the comment of the volume can be updated.

Parameters:
  • name – str The three-level (fully qualified) name of the volume

  • comment – str (optional) The comment attached to the volume

  • new_name – str (optional) The new name for the volume. Set it to rename the volume.

  • owner – str (optional) The user or group that owns the volume. The user who creates a volume becomes its owner; the owner can be changed afterwards with an update.

Returns:

VolumeInfo