Tracking deployment metrics on UbiOps¶

When you have deployed a model to production with UbiOps, you might want to track some custom metrics over time for a performance indication. This how to guides you through the process of creating a new custom metric and logging data to it from your deployment code. We cover the following steps

1. Define a new metric in UbiOps
2. Adjust your deployment code to log datapoints for this metric.
3. Tracking your metric in the WebApp

Defining a new metric in UbiOps¶

Before you can track any custom metric in UbiOps, the metric needs to be defined. You can either do this in the WebApp by navigating to Monitoring > Metrics > Create new metric, or via code. Let's walk through both.

When you click Create new custom metric in the WebApp, you will be redirected to a form where you need to provide the following information:

• Name: the name of your custom metric. It always has to begin with custom.. This is the name that will be used for the titles of associated graphs later. In our example we can call our metric custom.example
• Description (optional): the description of your custom metric.
• Unit (optional): the unit of measurement for your custom metric In our case we can use count.
• Metric type: how this metric should be processed. We support Gauge and Delta metrics (see metrics for more details). For our example we can use Gauge.
• Metric level (referred to as labels in the API): on what level do you plan to store the metric? In our case we need the metric on deployment level.

When you click Create the metric will be created and you can start logging data to it. Alternatively, you can also do the same with the Python client library:

import ubiops

configuration = ubiops.Configuration()
# Configure API token authorization
configuration.api_key['Authorization'] = "Token <YOUR_API_TOKEN>"
api_client = ubiops.ApiClient(configuration)
core_api = ubiops.CoreApi(api_client)

project_name = 'project_name_example' # str
data = ubiops.MetricCreate(
    name = 'custom.example',
    description = 'An example metric',
    metric_type = 'gauge',
    unit = 'count',
    labels = ['deployment_version_id']
)

# Create the metric
api_response = core_api.metrics_create(project_name, data)
print(api_response)

Logging datapoints from your deployment code¶

Now that the metric has been created in UbiOps, we can start logging data to it from our deployment code. In essence you just need to start up a UbiOps metric client, and use it's log_metric function.

from ubiops.utils.metrics import MetricClient

metric_client = MetricClient(project_name = "your_project_name")
metric_client.start()
metric_client.log_metric(
  metric_name = "custom.example",
  labels = {"deployment_version_id": "your_deployment_version_id"},
  value = your_metric_value
)

Let's have a look at how this would work with in your deployment code.

from ubiops.utils.metrics import MetricClient

class Deployment:

    def __init__(self, base_directory, context):
        print("Initialising deployment")
        self.metric_client = MetricClient(project_name=context["project"])
        self.metric_client.start()
        self.context = context

    def request(self, data, context):
        print(f"Processing request {context['id']}")

        example_value = 1

        # log the metrics to UbiOps
        print("Logging metrics...")
        self.metric_client.log_metric(metric_name="custom.example", labels={"deployment_version_id": self.context["version_id"]}, value=example_value)        

        return {}

The project name and the deployment version id can be retrieved from the context parameter passed in the init function, and used to log the metric value correctly.

Viewing your data¶

When you define your metric and adjust your code to track it as defined in the previous steps, you will be able to see your custom metric data in the UbiOps WebApp. Simply navigate to your deployment version details page and click on the Metrics tab. The graph in the WebApp will have a resolution of maximum one value per minute.