Skip to content

Working with files

UbiOps supports the use of files as input or output of both deployments and pipelines. The file datatype can be used to handle images, data files or other file objects in requests.

Managing files with buckets

Files are organized inside buckets. UbiOps has a file system that allows you to either create storage buckets directly on UbiOps, or connect to your own storage buckets on either Google, AWS or Azure. You can have multiple buckets per project and by default we provide you with a default bucket. Every project member and every deployment/pipeline in the project has read and write access to the default bucket.

Every file on UbiOps can be re-used in multiple requests in the same project, or used as intermediate storage for passing files between deployments in a pipeline.

Creating buckets

New storage buckets can be created on project level. In the WebApp you can create a new bucket on the storage page, that you can find via the sidebar. On the storage page you can click Create new bucket to either create a new UbiOps hosted bucket, or to connect to a bucket in your own cloud. This policy dictates the time period after which files in your bucket are automatically deleted. The default deletion policy is a week.

To connect your existing storage bucket in your own cloud environment, check the respective how-to's:

Managing bucket permissions

You can set more granular permissions on buckets that are not the default bucket.

  • Project members
  • Service users (API tokens)
  • Deployments

Permissions can be granted by assigning files related roles. There are four default roles for working with files:

  • files-reader: this role has read only permissions for files
  • files-writer: this role can read and write files
  • files-reader-restricted: this role has read only permissions and cannot list files. If users interacting with your deployment should only be able to view the specific files related to a request they made, you should use this role. This role is particularly useful for publicly exposed deployments.
  • files-writer-restricted: same as above, but then also with write permissions.

Deployments need to be explicitly granted permissions to read from and write to buckets. By default, deployments can only access the 'default' bucket. In case that your deployment has an input/output field of datatype 'file' and you want to access another bucket, make sure to grant the deployment permissions to do so.

For more information on assigning roles, see the permissions page.

Uploading new files and managing existing ones

Files are uploaded, listed, downloaded or deleted in UbiOps using an API request, either directly or using the WebApp, CLI or Python client library. In the WebApp you can browse through existing files, upload new files, or organize them in folders.

To support the process of uploading and downloading files, we have create a Python module named 'ubiops.utils'. It contains two methods: 'ubiops.utils.download_file' and 'ubiops.utils.upload file'. These methods can be used to do download or upload a file in one step. The following example uses an API Client api_client to upload a file file to the location files/file in bucket my-bucket in project 'my-project'

from ubiops import utils
# Upload a file

file_uri = ubiops.utils.upload_file( 
    client = api_client, #a UbiOps API client 
    project_name ="my-project", 
    file_path = "file", 
    bucket_name = "your-bucket", 
    file_name = "files/file" 
)
Alternatively, you can use the API directly to download or upload files in a two-step process that uses "signed URLs' When using the API directly to download or upload files, please note that it is a two-step process that uses "signed URLs".

For downloading: - Request a signed URL using the "Download a file" API endpoint. - Download the file using a standard HTTP GET request.

For uploading: - Request a signed URL using the "Upload a file" API endpoint. - Use the signed URL to upload the file using a HTTP PUT upload request. An example using cURL is: curl -X PUT <signed URL> --upload-file file.ext. - Azure buckets require two additional request headers to be sent with the upload, see the "Upload a file" endpoint on our Swagger documentation for more information.

A signed URL is valid for 5 minutes. Note that anyone in possession of the URL can upload or download the file during the 5-minute window.