Compute & scaling settings¶

UbiOps offers fine-grained settings to be able to configure your deployment according to your scaling and availability needs. You can control how many instances of your deployment are running at any time with the amount of resources that you configure. The scaling setting for training experiments are fixed.

The default settings are set to the following:

Maximum and minimum number of instances¶

UbiOps allows you to specify the maximum and minimum number of active deployment instances concurrently running. You can choose a value when creating and updating a deployment version by entering the desired value field in the dropdown list Advanced parameters.

This setting enables you to manage the resources used by the UbiOps auto-scaler. UbiOps will analyze the deployment workload and automatically decide to increase/decrease the number of available instances for the served deployment.

If you set the Minimum number of instances to 0, your deployment instances will be scaled down to 0 when no requests are sent to the deployment and the maximum idle time is reached (see the Maximum idle time section for more information). When the deployment instance is scaled to 0, it will require a cold start time for your first request. Cold start stands for your deployment instance to be started up and become ready to handle requests.

For a value of the Minimum number of instances greater than 0, UbiOps will always keep an active instance of your deployment, meaning that there won't be a cold start for your first request. However, scaling up to more than the minimum number of instances that you specified might still result in a cold start.

If you want to disable the auto-scaling feature, you can set the same value for the maximum and minimum number of instances. For example, if you set both the Minimum number of instances and the Maximum number of instances to 3, you will always have 3 active deployment instances and auto-scaling will not be applied. This will also overrule the Maximum idle time, i.e., after the maximum idle time has elapsed with no deployment requests you will still have 3 active instances.

For batch requests, UbiOps will perform concurrent deployment requests based on the maximum number of instances.

Maximum idle time¶

The Maximum idle time indicates the time (in seconds) after which the number of active deployment instances is scaled down to the value specified in Minimum number of instances, if no requests are performed. If the Minimum number of instances is set to 0, the deployment instance will shut down after the Maximum idle time has expired, resulting in a cold start for the next request. The Maximum idle time can be overruled by setting the Minimum number of instances to a value higher than 0. For example, if you set the Minimum number of instances to 1, UbiOps will keep 1 active instance of your deployment, even after the maximum idle time is reached.

Instance Type¶

Instance types determine the memory, vCPU, and storage allocation for your deployment. CPU allocation automatically scales with memory, with 1 vCPU core per 4 GiB of memory. This is a hard limit, so the memory usage of the deployment cannot exceed the limit defined by the instance type. Local storage scales with memory as well, 4 GiB per 1 GiB of memory. This means deployments can write to a local directory, but not unlimited. The local directory is not preserved over deployment restarts. UbiOps also supports instance types with GPUs, see GPU Deployments.

The following instance types are available:

You can choose a value when creating and updating a deployment version by picking the desired instance type from the drop-list in the Scaling and resource settings in the UbiOps WebApp.

Limits¶

The limits on the previously described settings are as following: