Skip to content

Data Connectors

Deployments can also be used to connect to external data sources. We provide template deployment packages written in Python for connecting to databases, cloud services and other software systems. These templates can be used without modifications, but can also be customized for more advances use cases. All connector templates are published on our Github.

Deploying connector templates

You can get started with connectors immediately by downloading one of the template deployment packages, and uploading it to UbiOps as you would do with any normal deployment. Create a deployment and first version for the connector and upload the ZIP as the code of the deployment.

The settings of the connector can be configured through environment variables. The list of environment variables that need to be configured is included in the README of each connector template. For security reasons, we advise you to configure environment variables containing passwords and other secrets as secret variables, to make them unreadable.

You can verify that your connector is working by making a direct manual request to the connector deployment, before including it in a pipeline.

Make sure to whitelist UbiOps in your firewall

Outbound connections from UbiOps will be made through a proxy with the following IPv4 address: 34.90.157.84. Make sure your firewall whitelists this IP address if necessary.

Customized or new connectors

For advanced use cases you can customize the template connectors. This can for example be useful if you want to do data processing before pushing your data to an external system, or if you want to include logic to update existing data instead of only inserting.

In addition to the existing template connectors, we published a basic structure that is used as the base for all connectors. Using this blueprint you can implement your own connector for a storage system that is not yet available as a template. UbiOps is frequently implementing new connectors as well, so the library of available connectors will expand in the future.

Available connector templates

The following connector templates are currently available on the UbiOps Github. The connectors are divided into output connectors, that can be used to push data from UbiOps into a database or other storage system, and input connectors, that can be used to retrieve data.

Each of the connectors contains a README with usage instructions.

Output connectors

Connector Deployment package Link
Amazon Web Services Kinesis ZIP Github
Amazon Web Services S3 ZIP Github
Azure Blob Storage ZIP Github
Google Cloud Platform BigQuery ZIP Github
Google Cloud Platform Cloud Storage ZIP Github
InfluxDB ZIP Github
Microsoft SQL server ZIP Github
MySQL ZIP Github
PostgreSQL ZIP Github

Input connectors

Connector Deployment package Link
RabbitMQ ZIP Github