Skip to content

Connectors

Connectors can be used to insert the output data from your Models to your own databases or other storage solutions. To do this, you can use a pipeline in which a model is connected to the connector.

  • Connectors for structured data insert each data point as a row in a table or similar structure.
  • Connectors of blob data insert each entry as a single object into the storage solution. We define a blob as any binary file.

Supported connectors

Connector Type
Amazon Web Services S3 Blob
Azure Blob Storage Blob
Google Cloud Storage Blob
Amazon Kinesis Structured
Google Cloud BigQuery Structured
MySQL Structured
PostgreSQL Structured

Structured/Blob vs. Structured/Plain

In our models we speak of structured and plain models, but for connectors we define structured and blob models. What is the difference exactly?

Essentially a structured model can output multiple fields, of which one (or more) can be of blob (file) type. Following this, a structured model can be linked to both a structured as blob connector: When attaching the connectors in a pipeline, you are asked to provide a mapping of the fields. This allows you to map the fields of the model to the structured connector. Moreover, the field of type blob of the structured model can be linked to the field blob of the blob connector.

Prerequisites

Before connecting UbiOps to your storage, make sure that:

  1. Your storage has the correct structure to store the data (such as a database and table).
  2. You have credentials and access rights are setup correctly.
  3. Your firewall is configured to allow access from UbiOps.

IP address of connectors

UbiOps connectors will query through a proxy with the following IPv4 address: 34.90.157.84. Make sure your firewall allows access to your storage from this IP address.

Creating connectors

In the left navigation bar, click on Connectors, on the connectors page then click CREATE.

Each connector requires the following fields to be set:

  • name : name of the connector.
  • type : type of the data store to connect to.
  • credentials : access credentials for the data store.
  • configuration : where in the data store the data should be written, for example, the table name.
  • fields : how the data store is organized (required only for structured connectors).

See Connector specific instructions for detailed instructions on each type of connector that UbiOps supports.

Connector credentials

The connectors authenticate with external systems using Credentials. UbiOps provides the ability to reuse existing credentials when creating new connectors. This way your login details only have to be entered once, and only one set of credentials need to be changed if something about the settings needs to be updated.

You can create these credentials during the creation of a Connector. Alternatively you can create credentials from the credentials overview page. You can go to this page by first clicking on Connectors on the left sidebar menu and then clicking on Credentials. On this page you can view, edit and create credentials. These credentials are matched to connector types.

See Connector specific instructions for detailed instructions on each type of credentials that UbiOps supports.