On the 25th of January 2024 we have released new functionality and made improvements to our UbiOps SaaS product. An overview of the changes is given below.
Python client library version for this release: 4.3.0
CLI version for this release: 3.3.0
Custom metrics
After many requests, we have added functionality for tracking your own custom metrics in UbiOps! You can now create and visualize custom metrics for training runs, deployments and pipelines. This way you can keep track of model specific metrics like accuracy, loss and precision.
To start tracking a new custom metric, it first needs to be defined in UbiOps. You can do so by navigating to Monitoring > Metrics > Create custom metric in the WebApp. After you’ve defined it, you can log metrics to it with the Python Client Library.
In the case of a training run you can adjust your training run as follows:
from ubiops.utils.metrics import MetricClient
def train(training_data, parameters, context):
training_run_id = context["id"]
project_name = context["project"]
metric_client = MetricClient(project_name=project_name)
metric_client.start()
# <YOUR TRAINING CODE>
example_value = 0
metric_client.log_metric(
metric_name = "custom.example", # Make sure you created a metric with this name beforehand
labels = {"request_id": training_run_id},
value = example_value
)
For a deployment you can do the following:
from ubiops.utils.metrics import MetricClient
class Deployment:
def __init__(self, base_directory, context):
self.metric_client = MetricClient(project_name=context["project"])
self.metric_client.start()
self.context = context
def request(self, data, context):
example_value = 0
metric_client.log_metric(
metric_name = "custom.example", # Make sure you created a metric with this name beforehand
labels = {"deployment_version_id": self.context["version_id"]},
value = example_value
)
Check out the documentation for all the details and example code snippets.
A customizable monitoring dashboard
We also made the monitoring dashboard in the WebApp (under Monitoring > General) customizable so you can decide what you want to see. You can combine both custom and default metrics in this view.
A dark theme for the WebApp
Do you prefer working in a dark theme? It is now possible to adjust the them of the WebApp. In the page header you will find a small sun/moon icon with which you can toggle between the themes.
Updated widgets for the request interface
We have added an image drawer and an image preview widget for file type input/output to the request interface. You can configure this on the Use deployment tab of your deployment.
Improved usage charts
We changed our organization and project usage charts to stacked bar charts so you can better see where your usage is coming from. On your subscription page you can see the organization usage chart, where the usage is split between the different projects. On the project settings page you can find a project usage chart where you can see which deployments are contributing to your credit usage.
Expanded the info in the context parameter for training runs
The training run context parameter now also contains the key project which provides the name of the project the code is currently being run in. This way the project parameter doesn’t need to be hardcoded when using the UbiOps client library from within the code. We also added the key experiment which contains the experiment name.
