Use files for persistent storage¶

In UbiOps the storage of deployments is not persistent. Deployments are stateless to ensure efficient scaling on our platform. Locally stored files are wiped from the deployment's memory after the instance shuts down. In this how-to, we show how you can use our file storage for persistent storage. This can be helpful for example when you want to have multiple objects in a pipeline reading from and writing to the same file.

We will first show how to upload files to UbiOps, and then show how we can use that file elsewhere.

Uploading files¶

Assume that we have created a test file during a request, and that we have stored it locally inside in our deployment instance. In the deployment code we can upload that file to a specific storage bucket by using the client library:

ubiops.utils.upload_file(api_client, project_name, file_input, bucket_name, file_name='testfile.txt')

Where api_client is connection to our api (that we have e.g. set-up in the __init__ of the deployment) with sufficient permissions to write files to our bucket of choice.

It is also possible to just pass the file as an output of the deployment. In that case you don't need to use the python client but you can simply return the file path to the local file in the return statement. See working with files for more information.

Using the file in the next deployment in a pipeline¶

There are multiple ways to pass this file to the next pipeline object. The exact way depends on your situation and your needs. I will describe two, direct passing, and indirect.

Direct passing¶

If your deployment object returns the file, and the next deployment in your pipeline expects it as input, you can just create a connection between the two. In that case the destination object will have access to the same file returned by the first object and can access it as follows in the request method:

testfile = data["testfile"]

Indirect passing¶

If the deployments are not connected in a pipeline, but you still want to use the same file inside another deployment, we can retrieve it via it's file URI, or via a combination of the file name and the bucket name where it's stored:

ubiops.utils.download_file(
    api_client,
    project_name,
    bucket_name,
    file_name='testfile.txt',
    output_path='.',
    stream=True,
    chunk_size=8192
)

Now you have access to the contents of the file in your deployment instance.

For more information on working with files, see working with files