Working with files¶
UbiOps supports the use of files as input or output of both deployments and pipelines. The file
datatype can be used to handle images, data files or other file objects in requests.
Managing files with buckets¶
Files are organized inside buckets. UbiOps has a file system that allows you to either create storage buckets directly on UbiOps, or connect to your own storage buckets on either Google, AWS or Azure. You can have multiple buckets per project and by default we provide you with a default
bucket. Every project member and every deployment/pipeline in the project has read and write access to the default bucket.
Every file on UbiOps can be re-used in multiple requests in the same project, or used as intermediate storage for passing files between deployments in a pipeline.
Creating buckets¶
New storage buckets can be created on project level. In the WebApp you can create a new bucket on the storage page, that you can find via the sidebar. On the storage page you can click Create new bucket to either create a new UbiOps hosted bucket, or to connect to a bucket in your own cloud. This policy dictates the time period after which files in your bucket are automatically deleted. The default deletion policy is a week.
To connect your existing storage bucket in your own cloud environment, check the respective how-to's:
- How to connect to an existing Google Cloud Storage bucket
- How to connect to an existing Amazon S3 bucket
- How to connect to an existing Azure blob storage bucket
Managing bucket permissions¶
You can set more granular permissions on buckets that are not the default
bucket.
- Project members
- Service users (API tokens)
- Deployments
Permissions can be granted by assigning files related roles. There are four default roles for working with files:
files-reader
: this role has read only permissions for filesfiles-writer
: this role can read and write filesfiles-reader-restricted
: this role has read only permissions and cannot list files. If users interacting with your deployment should only be able to view the specific files related to a request they made, you should use this role. This role is particularly useful for publicly exposed deployments.files-writer-restricted
: same as above, but then also with write permissions.
Deployments need to be explicitly granted permissions to read from and write to buckets. By default, deployments can only access the 'default' bucket. In case that your deployment has an input/output field of datatype 'file' and you want to access another bucket, make sure to grant the deployment permissions to do so.
For more information on assigning roles, see the permissions page.
Uploading new files and managing existing ones¶
Files are uploaded, listed, downloaded or deleted in UbiOps using an API request, either directly or using the WebApp, CLI or Python client library. In the WebApp you can browse through existing files, upload new files, or organize them in folders.
To support the process of uploading and downloading files, we have create a Python module named 'ubiops.utils'. It contains two methods: 'ubiops.utils.download_file' and 'ubiops.utils.upload file'. These methods can be used to do download or upload a file in one step. The following example uses an API Client api_client
to upload a file file
to the location files/file
in bucket my-bucket
in project 'my-project'
from ubiops import utils
# Upload a file
file_uri = ubiops.utils.upload_file(
client = api_client, #a UbiOps API client
project_name ="my-project",
file_path = "file",
bucket_name = "your-bucket",
file_name = "files/file"
)
For downloading: - Request a signed URL using the "Download a file" API endpoint. - Download the file using a standard HTTP GET request.
For uploading: - Request a signed URL using the "Upload a file" API endpoint. - Use the signed URL to upload the file using a HTTP PUT upload request. An example using cURL is: curl -X PUT <signed URL> --upload-file file.ext
. - Azure buckets require two additional request headers to be sent with the upload, see the "Upload a file" endpoint on our Swagger documentation for more information.
A signed URL is valid for 5 minutes. Note that anyone in possession of the URL can upload or download the file during the 5-minute window.