Pipelines allow users to define sequences of Deployments, by connecting the output of one deployment to the input of another deployment. This is useful when your application depends on a series of separate data transformations that need to operate in sequence.
UbiOps manages the dataflow between the deployments, so there is no need to manually make separate deployment requests. A deployment that is part of a pipeline is called a pipeline object.
You can use a pipeline just like you would use a deployment. All deployments can be used in multiple pipelines at once, or multiple times in a single pipeline, which provides a lot of flexibility. All objects in a pipeline will scale independently of each other, efficiently distributing the compute load for your process. Each object can use its own dependencies and programming language independent of the other objects in the pipeline.
Creating a pipeline¶
A new pipeline is created by going to Pipelines in the left navigation panel and then clicking the Create button. The input sent to the pipeline will be send to the first deployment in that pipeline. For more info about inputs, check out the In- and output pages. The deployment output connected to the pipeline end will be part of the response of a pipeline request. As with deployments, you can assign labels to pipelines, making it easier to filter them later on. If you need help building your first pipeline into UbiOps, see our pipeline starter tutorial.
As is the case with deployments, there can be multiple versions of a pipeline.
You can mark a version of a pipeline as the default version of that pipeline, which allows not having to specify the version when making a request to a pipeline. If left unspecified, the request will always go to the default version that is currently set.
There can only be one default version per pipeline at a time. If the default version is changed, it is effective immediately, meaning that any request after this moment will be directed to the new version. Note that already created but unprocessed batch requests will still be processed by the version that was the default when creating the batch request. Click on Edit in the Pipeline details page in the WebApp to change the default version of that pipeline.
The first version of a pipeline is always marked default.
To make a request to the default version, navigate to the Pipeline details page in the WebApp and click on Create Request.
See Monitoring for information on how to monitor your pipeline.
Referencing a default pipeline version
The default version of a pipeline can be used in a direct and batch request and when creating a request schedule.
Adding objects to a pipeline version¶
When you view the details of a pipeline version, by clicking on pipeline name, you enter the scope of the pipeline. In the General tab, the visual representation of the pipeline can be seen.
The Objects menu on the right gives an overview of all deployments in your pipeline. After pipeline version creation, this tab is empty. You can add new objects by clicking on Add new object in the General tab.
Each object is a reference to a deployment, which means that the deployment should be created first, before it can be used in a pipeline.
When you are adding an object you have to fill in the following fields:
|Reference name||The name of the deployment you want to use.|
|Reference version [OPTIONAL]||The version of the deployment you want to use. If not given, the default version of the deployment will be used.|
|Name||The name you want to give to this object. It must be unique within the pipeline.|
After creating a reference, the object should appear in the Objects menu as well as in the visual representation of the pipeline.
To connect objects in your pipeline, click on Create a connection. Here, you can specify the destination object for which you will specify the incoming connections.
One or more source objects can be used. A source output field can be connected to a destination input field if the data types match. All input fields of the destination object must be provided in the mapping, while the selection of output fields from the source object(s) can also be a subset. Multiple input fields of the destination object may be filled by the same source object output field.
To connect an object to the entry point of your pipeline, you need to select
pipeline_start as source object. To connect an object to the end/output of your pipeline, you need to select
pipeline_end as destination object.
Deleting objects in a pipeline¶
You can delete an object by clicking on it and then clicking on the trash icon. This will automatically delete all the attachments connected to this object too. Only the reference objects of the pipeline will be deleted, not the actual deployments that are being referenced.