Skip to content

Pipelines

Pipelines allow users to define sequences of Deployments, by connecting the output of one deployment to the input of another deployment. This is useful when your application depends on a series of separate data transformations that need to operate in sequence.

UbiOps manages the dataflow between the deployments, so there is no need to manually make separate deployment requests. A deployment that is part of a pipeline is called a pipeline object.

You can use a pipeline just like you would use a deployment. All deployments can be used in multiple pipelines at once, or multiple times in a single pipeline, which provides a lot of flexibility. All objects in a pipeline will scale independently of each other, efficiently distributing the compute load for your process. Each object can use its own dependencies and programming language independent of the other objects in the pipeline.

Creating a pipeline

A new pipeline is created by going to Pipelines in the left navigation panel and then clicking the CREATE button. The input sent to the pipeline will be send to the first deployment in that pipeline. For more info about inputs, check out the Deployments pages. As with deployments, you can assign labels to pipelines, making it easier to filter them later on.

Adding objects to a pipeline

When you view the details of a pipeline, by clicking on pipeline name, you enter the scope of the pipeline. In the General tab, the visual representation of the pipeline can be seen.

The Objects menu on the right gives an overview of all deployments in your pipeline. After pipeline creation, this tab is empty. You can add new objects by clicking on Add new object in the General tab. Each object is a reference to a deployment, which means that the deployment should be created first, before it can be used in a pipeline.

When you are adding an object you have to fill in the following fields:

Field Description
Reference name The name of the deployment you want to use.
Reference version The version of the deployment you want to use.
Name The name you want to give to this object. It must be unique within the pipeline.

After creating a reference, the object should appear in the Objects menu as well as in the visual representation of the pipeline.

Connecting objects

To connect objects in your pipeline, click on Connect two objects. Here, you can specify the source and destination object of the connection.

Drag and drop

You can also drag a line from one object to another to connect them. This will result in a connection creation form in which the source and destination are automatically set.

Two objects can be connected if the data type of the output field of the source object matches the data type of the input field of the destination object. All input fields of the destination object must be provided in the mapping while only the necessary fields of the source object might be selected. Multiple input fields of the destination object may be filled by the same source object output field.

To connect an object to the entry point of your pipeline, you need to select your pipeline as the source object.

Currently, simultaneous input from two objects into one is not supported, meaning that two source objects cannot provide part of the destination object input fields.

Deleting objects in a pipeline

You can delete an object by clicking on it and then clicking on the trash icon. This will automatically delete all the attachments of this object too. Only the reference objects of the pipeline will be deleted, not the actual deployments that are being referenced.