Bucket & file operations in UbiOps using Python¶

In this tutorial we will try different ways of interacting with buckets and files operations in UbiOps through Python. This will be done through:

making HTTP requests to our API endpoints
calling the API endpoints via the methods of our Python Client Library
calling methods from utils

!pip install -U ubiops

We will first use HTTP requests on all endpoints for buckets and files. We need to set up some variables below. Before that you need to have created a project and an API token for it. This is how you do it. The token needs project-editor permission.

API_TOKEN = f"Token "  # Make sure this is in the format "Token token-code"
PROJECT_NAME = ""  # You can name your UbiOps project however you want, but it must be globally unique and created in advance

HOST = "https://api.ubiops.com/v2.1"  # Leave this one as is. It will be used to send requests.
HEADERS = {
    "Authorization": API_TOKEN,
}

Let us start by using the Files endpoints of our API. These can be studied further on our Swagger page. We will focus first on buckets. We will first list all buckets within this project.

import requests


response = requests.get(
    url=f"{HOST}/projects/{PROJECT_NAME}/buckets",
    headers=HEADERS,
)

print(response.json())

As we can see, there is only the default bucket now, unless you already created a new bucket, or coupled an existing bucket. We can create new buckets, we can give them labels and descriptions.

for i in range(3):
    response = requests.post(
        url=f"{HOST}/projects/{PROJECT_NAME}/buckets",
        headers=HEADERS,
        data={
            "name": f"bucket-{i + 1}",
            "provider": "ubiops",
            "credentials": {},
            "configuration": {},
            "labels": {
                "type": "bucket",
            },
            "description": "My bucket description",
        },
    )

    print(response.json())

Then we can see the details of a bucket of our choice.

bucket = "bucket-1"

response = requests.get(
    url=f"{HOST}/projects/{PROJECT_NAME}/buckets/{bucket}",
    headers=HEADERS,
)

print(response.json())

We can update the details of a bucket.

response = requests.patch(
    url=f"{HOST}/projects/{PROJECT_NAME}/buckets/{bucket}",
    headers=HEADERS,
    data={
        "description": "My bucket updated description",
    },
)

print(response.json())

We will now delete this bucket.

response = requests.delete(
    url=f"{HOST}/projects/{PROJECT_NAME}/buckets/{bucket}",
    headers=HEADERS,
)

print(response)

Now that we have gone through bucket endpoints, we can focus on files. Let's create and add some files to one of our buckets. We need to first call our endpoint to get a signed URL, then we use that URL to upload the file to the bucket. A signed URL grants temporary and restricted access for making a request. It includes authentication details within its query string, enabling users who lack credentials to carry out certain tasks on a resource. More about this here.

!mkdir files

%%writefile files/file_1.txt
Hello

%%writefile files/file_2.txt
World!

bucket = "bucket-2"

for i in range(2):
    # Get signed url from Google Cloud Storage to upload the file
    response = requests.post(
        url=f"{HOST}/projects/{PROJECT_NAME}/buckets/{bucket}/files/file_{i + 1}",
        headers=HEADERS,
    )

    print(response.json())

    # Make PUT request to that url to upload the file to UbiOps
    response = requests.put(
        url=response.json()["url"],
        data=open(file=f"files/file_{i + 1}.txt"),
    )

    print(response)

We can now see the files in the bucket.

response = requests.get(
    url=f"{HOST}/projects/{PROJECT_NAME}/buckets/{bucket}/files",
    headers=HEADERS,
)

print(response.json())

We can get the details of a file

file = "file_2"

response = requests.get(
    url=f"{HOST}/projects/{PROJECT_NAME}/buckets/{bucket}/files/{file}",
    headers=HEADERS,
)

print(response.json())

And then delete it.

response = requests.delete(
    url=f"{HOST}/projects/{PROJECT_NAME}/buckets/{bucket}/files/{file}",
    headers=HEADERS,
)

print(response)

Now we will download the other file to our local environment.

file = "file_1"

# Get signed url from Google Cloud Storage to download the file
response = requests.get(
    url=f"{HOST}/projects/{PROJECT_NAME}/buckets/{bucket}/files/{file}/download",
    headers=HEADERS,
)

print(response.json())

# Make GET request to download the binary content of the file
response = requests.get(
    url=response.json()["url"],
)

print(response)

# Copy the file content from response to a new file which will be saved locally
if response.status_code == 200:
    with open(file=f"./{file}.txt", mode="wb") as file:
        file.write(response.content)

Now we will do the same steps as above but this time using Files API endpoints methods from the UbiOps Python client library, which can be found here. We first need to get the core API, and then we can start making calls to it.

import ubiops


configuration = ubiops.Configuration(host=HOST, api_key=HEADERS)
api_client = ubiops.ApiClient(configuration=configuration)
core_api = ubiops.CoreApi(api_client=api_client)
print(core_api.service_status())

We will first list all buckets within this project.

core_api.buckets_list(project_name=PROJECT_NAME)

We will create new buckets.

for i in range(3, 6):
    response = core_api.buckets_create(
        project_name=PROJECT_NAME,
        data={
            "name": f"bucket-{i + 1}",
            "provider": "ubiops",
            "credentials": {},
            "configuration": {},
            "labels": {
                "type": "bucket",
            },
            "description": "My bucket description",
        },
    )

Then we can see the details of a bucket of our choice.

bucket = "bucket-4"

core_api.buckets_get(project_name=PROJECT_NAME, bucket_name=bucket)

We can update the details of a bucket.

response = core_api.buckets_update(
    project_name=PROJECT_NAME,
    bucket_name=bucket,
    data={
        "description": "My bucket updated description",
    },
)

print(response)

We will now delete this bucket.

core_api.buckets_delete(project_name=PROJECT_NAME, bucket_name=bucket)

Let's add some files to one of our buckets.

bucket = "bucket-3"

for i in range(2):
    # Get signed url from Google Cloud Storage to upload the file
    response = core_api.files_upload(
        project_name=PROJECT_NAME, bucket_name=bucket, file=f"file_{i + 1}"
    )

    # Make PUT request to that url to upload the file to UbiOps
    response = requests.put(
        url=response.url,
        data=open(file=f"files/file_{i + 1}.txt"),
    )
    print(response)

We can now see the files in the bucket.

core_api.files_list(project_name=PROJECT_NAME, bucket_name=bucket)

We can get the details of a file

file = "file_2"

core_api.files_get(project_name=PROJECT_NAME, bucket_name=bucket, file=file)

And then delete it.

core_api.files_delete(project_name=PROJECT_NAME, bucket_name=bucket, file=file)

Now we will download the other file to our local environment.

file = "file_1"

# Get signed url from Google Cloud Storage to download the file
response = core_api.files_download(
    project_name=PROJECT_NAME, bucket_name=bucket, file=file
)
print(response)
# Make GET request to download the binary content of the file
response = requests.get(
    url=response.url,
)

print(response)

# Copy the file content from response to a new file which will be saved locally
if response.status_code == 200:
    with open(file=f"./{file}.txt", mode="wb") as file:
        file.write(response.content)

Now we will take a look at the ubiops.utils methods for files, which can be found here. First there is file uploading. We will put one file inside a folder and then the second one will be directly in the bucket. UbiOps bucket will create the folder for us if we specify it before our file name. Note that folders can also be created with the core_api.files_upload(), in the file parameter, and with the HTTP request for file upload, just add the folder name previous to the file name.

bucket = "bucket-5"

help(ubiops.utils.upload_file)

ubiops.utils.upload_file(
    client=api_client,
    project_name=PROJECT_NAME,
    file_path=f"files/file_1.txt",  # Path to our local file
    bucket_name=bucket,
    file_name=f"files_folder/file_1", # How we want it to be named in the bucket, and under which folder(s)
)

ubiops.utils.upload_file(
    client=api_client,
    project_name=PROJECT_NAME,
    file_path="files/file_2.txt",
    bucket_name=bucket,
    file_name="file_2",
)

We can now see the uploaded files. Let's select to see only the one within the files_folder using the prefix parameter.

help(core_api.files_list)

core_api.files_list(
    project_name=PROJECT_NAME, bucket_name=bucket, prefix="files_folder"
)

Then there is file downloading. We can download the first file, the one in the folder.

ubiops.utils.download_file(
    client=api_client,
    project_name=PROJECT_NAME,
    bucket_name=bucket,
    file_name="files_folder/file_1",  # File path in the bucket, folder needs to be specified here
    output_path=f"./file_1.txt", # Where to download it on our local system and how to name it
)

So that's it! We have gone through all the ways of interacting with UbiOps files and buckets. There are more how-to's about storage on our docs page, for example this one will teach you how to achieve persistent storage with UbiOps.