Skip to content

Pipeline deployment quickstart

This tutorial describes the steps to create and deploy a Pipeline in UbiOps. With pipelines, a user can set up a data flow through a sequence of Deployments from a single entry point. Read the Pipelines page for more information.

In this tutorial, a pipeline will be created for weather temperature prediction. The first deployment in the pipeline performs an API request to retrieve publicly available KNMI data. The second deployment predicts the temperature of the next hour, using a moving average. Finally, there is the possibility to create a table in your database (for example, a PostgreSQL or MySQL database) and to include a connector to this database in the pipeline.

In this tutorial you will

  • Create two deployments
  • Use environment variables
  • Create a pipeline
  • Add objects to the pipeline
  • Perform a pipeline request
  • Create a deployment to connect to a database
  • Perform a pipeline request and see the results in the database

Create deployments with environment variables

Create two deployments by using the following specifications. If you are not familiar with creating a deployment, please read the Deployment quickstart first. In the environment variables section in deployment creation form in the WebApp, environment variables can be assigned to the deployment. For more information on environment variables, see Environment variables.

Go to UbiOps and login. In the navigation bar go to Deployments and click on CREATE. Create the following two deployments:

Deployment 1:

Create the deployment:

  • Name: knmi-data-collector
  • Description: This deployment collects the temperature per hour from the publicly available KNMI API.
  • Input fields (structured): end_date (string)
  • Output fields (structured): data (array of strings)

Click Next step, and create the version, environment variables and upload the code:

  • Version name: v1
  • Language: Python 3.7
  • Zip-file: download zip
  • Deployment mode: Express
  • Request retention and retention time: Full and 1 week (default)
  • Deployment environment variables:

    • The API URL:

      • Variable name: API_URL
      • Value: https://www.daggegevens.knmi.nl/klimatologie/uurgegevens
      • Secret: No (default)
    • The number of days in the future for which predictions are computed (optional, default 3):

      • Variable name: N_DAYS
      • Value: 1
      • Secret: No (default)

Click Create.

Deployment 2:

Create the deployment:

  • Name: temperature-predictor
  • Description: This deployment predicts the temperature of the next hour.
  • Input fields (structured): data (array of strings)
  • Output fields (structured): forecast (double precision)

Click Next step, and create the version, environment variables and upload the code:

  • Version name: moving-average
  • Language: Python 3.7
  • Zip-file: download zip
  • Deployment mode: Express
  • Request retention and retention time: Full and 1 week (default)
  • Deployment environment variables:
    • The number of hours that are used by the moving average (optional, default 3):
      • Variable name: WINDOW_SIZE
      • Value: 2
      • Secret: No (default)

Click Create.

The specified environment variables are now available in the code by importing the environ variable from the os library (from os import environ). If you followed the steps above, environ['API_URL'] and environ['N_DAYS'] can be used in the knmi-data-collector deployment package and environ['WINDOW_SIZE'] in the temperature-predictor deployment package. See the ZIP files for these two deployments provided above.

Overview and Update deployment environment variables

The deployment (version) environment variables can be found in the Environment variables tab on the deployment (version) details page. Environment variable can be specified on project, deployment and deployment version level and are inherited in this order, which means that project environment variables can be overwritten by deployment environment variables, which can be overwritten by deployment version environment variables.

If you followed the steps above, the environment variables were created on deployment level. This means that they will be available for all versions of those deployments.

Create pipeline

Now that the deployments are created, and the environment variables assigned, the pipeline can be created:

  1. Go to UbiOps and login.
  2. In the navigation bar go to Pipelines -> Create pipeline.
  3. Set the Name to weather-forecast.
  4. Set the Input type to structured.
  5. Set the Input fields:
    • Add an input field current_date with data-type string .
  6. Set the Output type to structured.
  7. Set the Output fields:
    • Add an output field forecast with data-type double precision .
  8. Click on the Create button.

Create a pipeline version

Create a first version of the pipeline:

  1. Go to UbiOps and login.
  2. In the navigation bar go to Pipelines.
  3. Click on the name of the pipeline weather-forecast.
  4. Click Create version.
  5. Set the Name to v1.
  6. Click on the Create button.

Add and connect pipeline objects

To add the two deployments to the created pipeline:

  1. Go to UbiOps and login.
  2. In the navigation bar go to Pipelines.
  3. Click on the name of the pipeline weather-forecast.
  4. Click on the name of the pipeline version v1.

Object 1

  1. Click on the Add new object button.
  2. Set the Reference name to knmi-data-collector.
  3. Set the Reference version to v1, or keep it empty if you want to use the default deployment version.
  4. Set the Name to knmi-data-collector.
  5. Click on the Create button.

Object 2

  1. Click on the Add new object button.
  2. Set the Reference name to temperature-predictor.
  3. Set the Reference version to moving-average, or keep it empty if you want to use the default deployment version.
  4. Set the Name to moving-average.
  5. Click on the Create button.

Attachment 1

  1. Click on the Create a connection button.
  2. Set the Source object to pipeline_start.
  3. Set the Destination object to knmi-data-collector.
  4. Set the Mapping.
    • Select current_date to be connected to end_date.
  5. Click on the Create button.

Attachment 2

  1. Click on the Create a connection button.
  2. Set the Source object to knmi-data-collector.
  3. Set the Destination object to moving-average.
  4. Set the Mapping.
    • Select data to be connected to data.
  5. Click on the Create button.

Attachment 3

  1. Click on the Create a connection button.
  2. Set the Source object to moving-average.
  3. Set the Destination object to pipeline_end.
  4. Set the Mapping.
    • Select forecast to be connected to forecast.
  5. Click on the Create button.

Pipeline request

Now that the pipeline has been deployed in UbiOps, it is ready to receive data. You can test the pipeline using either the UbiOps WebApp or the UbiOps API.

Pipeline request using the WebApp

You can test your newly created pipeline by going to its page.

  1. Click on Pipelines in the sidebar on the left and then click on your pipeline name. Then click on the pipeline version name.

  2. Now click CREATE REQUEST. Fill in the parameters:

    • The current_date parameter. You could set the current date by using the year-month-day format (for example, "2020-01-01"). If you leave the current date empty, the deployment will automatically select today as current date.

Click Create direct request to create the request.

  1. When the deployment has finished processing, the pipeline results are displayed. In addition, the results for each object in the pipeline are shown. Click on the row of an object to see its input and output. Note, this is only possible for the pipeline objects of which the deployment version has request_retention_mode set to "full".

Pipeline request using the API

API Token

For this part of the tutorial you will need an API Token with the correct permissions.

  1. Visit Create pipeline request.

    • Before making your first call click lock icon in the upper right corner, and fill in your UbiOps API Token in the format:

    Token 1ig03sggdqvqtvl2n7xc4q7npn7hqqe97od1qiyg (example)

    • Click on the Try it button in order to make a manual API request.
    • In the API explorer set all parameters for the manual pipeline insert:

    • The name of your project.

    • The name of your pipeline. Should be weather-forecast if you followed our example above.
    • The data should contain the current_date parameter. You could set the current date by using the year-month-day format (for example "2020-01-01"). If you set the current date to an empty string (""), the deployment will automatically select today as current date:
      {
      "current_date": ""
      }
      
    • Execute the API request by pressing Execute button, located at the bottom of the form.
  2. When the API call is successful, you will receive a server response code 200 with a dictionary containing:

  3. pipeline: The name of the pipeline.
  4. version: The name of the pipeline version.
  5. result: The request output of the pipeline.
  6. error_message: The error that the pipeline would generate if it were to fail.
  7. deployment_requests: A list of requests, with each:
    • id: The request id of the pipeline object, which can be used to filter logs or retrieve the deployment request.
    • pipeline_object: The name of the pipeline object.
    • deployment: The name of the deployment it references to.
    • version: The name of the deployment version it references to.
    • success: The state of the prediction request. Should be "true", which indicates that the deployment request finished.
    • error_message: The error that the deployment would generate if it were to fail. It should be "null" for this example.

(Optional) Create a table in your database

The next optional steps will describe the creation of a table in your own database that will be used to collect the results of the previous described weather forecast.

PostgreSQL Create a table in your PostgreSQL database containing a unique identifier, and a temperature_forecast column. We use in the example code below schema="weather_forecast" and table="pipeline_results". You can change this if you wish.

CREATE TABLE weather_forecast.pipeline_results
(
    id bigserial NOT NULL,
    temperature_forecast double precision NOT NULL,
    PRIMARY KEY (id)
)
WITH (
    OIDS = FALSE
);

MySQL Create a table in your MySQL database containing a unique identifier, and a temperature column. We use in the example code below table="weather_forecast". You can change this if you wish.

CREATE TABLE `weather_forecast` 
( 
  `id` SERIAL NOT NULL, 
  `temperature_forecast` DOUBLE NOT NULL 
);
You could also create a table or stream in, for example, Google Cloud BigQuery or Amazon Kinesis.

Giving UbiOps access To make UbiOps able to access your created database, you need to whitelist our IP address:

Whitelisting UbiOps in your firewall

Outbound connections from UbiOps will be made through a proxy with the following IPv4 address: 34.90.157.84. Make sure your firewall whitelists this IP address if necessary.

(Optional) Create a connector

Now that your table is created in your database and UbiOps has access to it, you can create a Deployment to be used as a database connector. For many database connections, including the ones mentioned above, you can use our Connector templates. You only need to zip the code and upload it to your own deployment. All our connector templates are published on our Github.

Create the deployment:

  • Name: weather-forecast-storage
  • Description: This deployment stores the weather forecast in a database.
  • Input fields (structured): temperature_forecast (double precision)
  • Output fields (structured): nothing

Assign the environment variables:

Assign all the environment variables that your deployment requires. If you use a connector template, you can find the required environment variables in the README. For example, PostgreSQL or MySQL.

Create the version and upload the code:

  • Version name: v1
  • Language: Select the programming language of your deployment.
    If you use a connector template, you can use Python 3.6 or higher.
  • Memory allocation: We recommend to allocate at least 512 MiB of memory to the deployment
  • Zip-file: Zip one of the deployment packages of our connector templates on our Github, or zip your custom code (see Deployment package structure for the required format).
  • Deployment environment variables: Assign all the environment variables that your deployment requires. If you use a connector template, you can find the required environment variables in the README. For example, PostgreSQL or MySQL.

After the creation, the deployment is build. Ones it is available, you can add it to your pipeline, or you could test it first by creating a request. For a tutorial on creating a deployment request, see Deployment quickstart. In the next steps, we will add the connector deployment to the weather forecast pipeline, and create a pipeline request.

(Optional) Add a connector to pipeline

The above described pipeline contains two objects: one that performs an API call to collect the KNMI data, and one that estimates the temperature of the next hour by calculating the moving average. This pipeline can be extended by the previously created connector. To do this, please follow the next steps.

Create a pipeline object

  1. Go to UbiOps and login.
  2. In the navigation bar go to Pipelines.
  3. Click on the pipeline name. Should be weather-forecast if you followed our example above.
  4. Click on the version name. Should be v1 if you followed our example above.
  5. Click on the Add new object button.
  6. Set the Reference name to weather-forecast-storage.
  7. Set the Reference version to v1, or keep it empty if you want to use the default deployment version.
  8. Set the Name to store-forecast.
  9. Click on the Create button.

Create an attachment

  1. Click on the Create a connection button.
  2. Set the Source object to moving-average.
  3. Set the Destination object to store-forecast.
  4. Set the Mapping.
    • Select forecast to be connected to temperature_forecast.
  5. Click on the Create button.

(Optional) Create pipeline request and see results in database.

To create a pipeline request, please follow the same steps described in Pipeline request.

When the request finishes, the results will be available in your database.