Building a low-code app powered by AI (with Mendix and UbiOps)

Low-code platforms such as Mendix are a great way to develop web & mobile applications in a fraction of the time that developing an application can normally take. With the help of such a platform, you don’t need to code your app line-by-line, but instead, construct it using pre-built elements and components. This makes building your own app way faster, more intuitive and easier to debug.

Despite the growing interest in data science and machine learning, most low-code platforms do not include functionality for AI and rely on integrations with other tools. What if you have a model available or are able to build one, and you want to turn it into an end-to-end application for your client? Such as an image recognition application, a chatbot or a recommendation system.

Figure 1: general architecture overview

In this article we will show you how you can do this via the example of an age estimation application. With this app you can upload a picture of yourself, a friend, or some random person from the internet and the app will estimate their age based on the person’s face. Pretty cool, right?

To do so we use a pre-trained (open source) neural network, and two tools: Mendix and UbiOps. Both Mendix and UbiOps can be used for free, so you can try it out yourself as well.

Let’s walk through it step-by-step.

Components & Overview

Image recognition model and training data

For the image classification itself, we will be using a pre-trained neural network implemented with the Caffe deep learning library. The age estimation model was developed at the ETH Zurich and trained on a publicly available image dataset scraped from IMDB and Wikipedia, containing faces of celebrities and public figures, as well as their age. 

For more details and information about the model you can visit: https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/  Reference: Rasmus Rothe and Radu Timofte and Luc Van Gool, Deep expectation of real and apparent age from a single image without facial landmarks, International Journal of Computer Vision, vol. 126, 2018

Serving infrastructure and front-end 

To build the application logic and front-end, we use the low-code platform Mendix. To deploy and run the Caffe neural network and use it as a service with an API endpoint we make use of UbiOps. The serving endpoint of the model in UbiOps can be consumed by the Mendix app to make requests (see figure 1). 

Why this setup? 

It doesn’t require any knowledge of how to set up the underlying IT architecture. We don’t have to worry about setting up servers, deploying our application, configuring networking, user management, scalability and uptime. Mendix and UbiOps are both SaaS services that take away the difficult work. Allowing us to create this app in no time! The figure below shows how everything comes together:

Figure 2: high level architecture

Figure 2: high level architecture 

Requirements

In order to get started you need the following:

  • A UbiOps account. You can create one for free at https://app.ubiops.com/sign-up.
  • A Mendix account. You can create one for free at https://signup.mendix.com/.
  • Mendix Studio Pro installed (only available on Windows, or via Parallel desktop or similar on mac).
  • The pre-trained Caffe deep learning model files. You can download them from: https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/ (scroll to the bottom)
    You need both the `.caffemodel` file as well as the `age.prototxt` file. We use the IMDB-WIKI one.
  • Basic knowledge of Python and a basic understanding of REST APIs.

Getting to work: Deploying the age estimation model 

First we will look at deploying the deep learning model on UbiOps so we can make requests to it.

After we upload our code & pre-trained models, UbiOps creates a Docker image with that code and all the necessary packages and dependencies included. After building, it’s available as a live service with a REST API endpoint that we can call from Mendix. This basically enables us to run any type of data processing code behind an endpoint and use it from wherever we want.

To deploy the pretrained Caffe model on UbiOps, we need to write some Python code first. UbiOps requires a deployment.py file with a request function in it. This is the function UbiOps will call every time a request is made through the serving endpoint.

We will use the `deployment.py` template available on the UbiOps Github and make a few edits to adjust it for our purpose.

Here is the final code of the `deployment.py`:

from setup_logging import setup_logging
import logging
import sys
sys.path.append('/usr/lib/python3/dist-packages') # We need to point to the location where Caffe installs its Python lib
import os
import cv2
import caffe
import numpy as np
from PIL import Image
import base64
import io

See the final code

def base64_to_image(enc_str):

   """

   Decodes a base64 string to an image and returns it as a Numpy array.

   The image will be resized using OpenCV to a resolution of 224x224 pixels.

   """

   dec_str = base64.b64decode(str(enc_str))

   img = Image.open(io.BytesIO(dec_str))

   img_arr = np.asarray(img)

   res = cv2.resize(img_arr, dsize=(224, 224), interpolation=cv2.INTER_CUBIC)


   return res


class Deployment:


   def __init__(self, base_directory):

       """

       Initialisation method for the deployment. This will be called at start-up of the model in UbiOps.


       :param str base_directory: absolute path to the directory where this file is located.

       """


       setup_logging()

       logging.info("Initialising Caffe model")


       # Here we initialize the Caffe classifier model with the pre-trained files

       model_def_file = os.path.join(base_directory, "age.prototxt")

       caffe_model = os.path.join(base_directory, "dex_chalearn_iccv2015.caffemodel")


       self.net = caffe.Classifier(model_def_file, caffe_model)


   def request(self, data):

       """

       Method for model requests, called for every individual request


       :param dict data: dictionary with the model data. In this case it will hold a key 'photo' with a base64 string

       as value.

       :return dict prediction: JSON serializable dictionary with the output fields as defined on model creation

       """


       logging.info("Processing model request")


       # Convert the base64 string input to a Numpy array of the right format. Using the function defined above.

       photo_data = base64_to_image(data['photo'])




       # Call the predict function of the Caffe net

       out = self.net.predict([photo_data], oversample=False)

       # From the output array we return the index of the largest value. The out array holds probabilities for

       # ages 1-100.

       age = out[0].argmax()


       # Here we return a JSON with the estimated age as integer

       return {'age': int(age)}

Some notes on the code:

  • The input of the request function is a string. Later, we will send the image from Mendix as a base64 encoded string to UbiOps. This is because Mendix can not send files from their REST request module.
  • The output of the request function is an integer for the estimated age.
  • We have added one function (base64_to_image) for transforming the image to a Numpy array and placed it outside of the Deployment class.

Within UbiOps, the zipped folder “deployment package” is used to upload our code to the platform so UbiOps can deploy everything. The structure of the zip is as follows (note that there is a parent folder inside the zip):

Figure 3: Contents of the deployment_package_caffe.zip file 

  • The deployment.py file (see code above) contains the request function that does the actual data handling and model inference in Python. 
  • A requirements.txt file that lists the required Python packages.
# This file contains package requirements for the model

# installed via PIP. Installed before model initialization
opencv-python==4.5.1.48

imageio==2.5.0
  • The downloaded Caffe model files (.caffemodel and .prototxt) that we refer to in the deployment.py file.
  • A `ubiops.yaml` file. This is used to tell UbiOps what OS level packages need to be installed in the Docker. We need this to install Caffe and its dependencies.
apt:

 packages:

   - caffe-cpu

   - ffmpeg

   - libsm6

   - libxext6
  • The setup_logging.py file is not mandatory, but used to integrate logs from the code with UbiOps.

Now we log in to UbiOps to deploy our model. With the finished deployment_package, we can create the first version of the deployment via the UbiOps UI (can also be done via the CLI or client library). 

First, we go to ‘Deployments’ in the menu on the left. There we click ‘Create’ and tell UbiOps a few things:

  • We give our model a nice original name like ‘mendix-age-estimation-app’

The Input and output type and fields that our model expects. You can see in the request() function in the ‘deployment.py’ that the variables are the following:
As ‘Input’ we define a variable ‘photo’ with as type ‘string’
As Output we define a variable ‘age’ with type ‘integer’

Figure 4: Deployment creation step in UbiOps 

Figure 5: Deployment creation step in UbiOps (bottom of page)

Figure 5: Deployment creation step in UbiOps (bottom of page)

We click “Next step” and there define the following:

  • Set the Python language to Python 3.8.
  • Click ‘Upload code’ and there select the zip file from our laptop to upload.
  • The rest we can leave as it is.

There are some useful advanced settings you can play with, but we don’t need them for now.

Figure 6: Uploading our code (deployment package) in UbiOps

Figure 6: Uploading our code (deployment package)  in UbiOps

Now we click “Create“ and UbiOps automatically starts building and deploying the model. Note that it might take a while for the deployment package to upload. It is around 500MB which can take some time depending on your internet connection. 

After upload, the status of the version changes to ‘Building’. We can actually follow what is happening in the background if we click on the version name and on the next page click the logo icon next to the status. You can see the building logs like this:

Figure 7: The logs in UbiOps from the building of the container

Figure 7: The logs in UbiOps from the building of the container

After a few minutes of building (installing Caffe takes a while), our age estimation model is ready to be used! You can test it if you want by clicking on the version and clicking ‘Create direct request’. Note that the model expects a base64 string of an image (so you need to convert the image first, this will be done by Mendix in the background).

Figure 8: The model is deployed and available for requests

Figure 8: The model is deployed and available for requests 

Our Caffe model is now running live and we can send data to it, great!

As the last thing, we need to create a service user with an API token in UbiOps so the Mendix app can authenticate with the UbiOps endpoint. You can do this in the Users & Permissions tab on the left. Make sure to copy the token and save it for later. To do so, assign the role of “project-admin” to the user of UbiOps.

Figure 9: Add a token and save it for later

Now we will switch to Mendix to create the front end and the connection to UbiOps. 

Building the Mendix front-end

To develop the front-end of our app we use Mendix Studio (webApp) and Mendix Studio Pro (desktop version). 

We will create a simple app which has three pages:

  1. a home page with a button to upload an image.
  2. a page to upload the image (pop up).
  3. a page to display the estimated age of the person in the picture.

Note: Later, we also added a fourth page that provides the user with more info and links to this article. The first three pages are connected via two microflows because of the underlying logic (more on that later). The latter page is stand-alone, and is activated by clicking on the “?” on the second page.

Step 1: Defining the Domain Model

As a first step, we create a “domain model” entity. This is basically the information/data model of the app. Our domain model is called “Photo” and has a ‘System.Image’ property so Mendix knows it includes an image, attached to this are two attributes called age and photo. In these attributes, we will later save the uploaded photo as a string, and the age as a response from ubiops as an integer. 

 Figure 10: Domain model in Mendix

Figure 10: Domain model in Mendix

Step 2: Creating the home page, the upload page and the microflow.

First we create a “form” page using one of the pre-built templates in Mendix. After some adjustments and adding a button that said “upload your picture here” (see figure 2), we add a “microflow” to trigger another page where one can actually upload and submit the picture (see figure 3). Mendix uses the so-called microflows to create the logic behind the pages. 

Figure 11: home page for the age estimation app (editing mode, studio view))

Figure 11: home page for the age estimation app (editing mode, studio view))

Figure 12: Pop up page with file upload widget (editing mode)

Figure 12: Pop up page with file upload widget (editing mode)

This is simply a page with a standard widget called “image uploader” and some lay-out customizations. You can drag and drop it onto the page. Not much coding to be done here. 

Clicking the button “upload your picture here” triggers the microflow that can be seen in figure 13. This microflow first creates an object in the entity “Photo” (see domain model), then opens the pop-up to upload a photo (see figure 12), and then closes the page once done. For more detail on microflows see step 3.

Figure 13: Microflow to open and close the “upload image” pop-up 

Step 3: Create microflow to call UbiOps API

Once a user clicks “submit” in figure 12, a new (and more advanced) microflow is triggered to encode the image, call the UbiOps API and return the response value. In this step, we explain how that microflow works. 

Figure 14: Overview of microflow to call the UbiOps API and return the result

Mendix must first “commit” the image, in other words, “save” it in the temporary database of our app. The next step is to encode the image into a string. This is required because Mendix cannot send files with the REST request functionality. So we used the “Base64 encoder” from the Community Commons Functions Library. The input is the entity Photo, whereas the output is a string stored in the variable “photo” attached to the entity. 

Figure 15: base64 encoder

This means that we now have a variable called “photo” in the Mendix database, with as value the base64 encoded string of the original image.

Now we are ready to set up the REST API post call to call the API endpoint of the deployed deep learning model in UbiOps. Figure 16 shows the Mendix ‘Call REST’ settings on http method (post), authentication, http headers, the request and the response. You can paste here the API URL of the model in UbiOps (starting with https://api.ubiops.com/v2.1/…) Going through all the details is beyond the scope of this article. 

Figure 16: REST API call to UbiOps details

Figure 17: HTTP Headers configuration for the Call REST module

In the HTTP Headers tab, we add the API token from UbiOps to the HTTP header for the Mendix request. The value should include the ‘Token’ keyword in the string as shown in the image.

It is important to note that the response from UbiOps is stored in a new variable called “age”. See figure 18 for the configuration. 

Figure 18: configuring the response from UbiOps and storing it in a new variable “age”.

Note that in the domain model (see Figure 20) you can already see “age” as a variable in the entity “Photo”. Storing the returned value for “age” is done in the “change photo” step. This is similar to the first “change photo” step (see the microflow in figure 14).

To correctly map the return value from UbiOps and store it in the variable “age”, we create an import mapping. This will select the right variable from the HTTP response. The JSON format as configured in Mendix is shown here: 

Figure 19: JSON format as configured in Mendix

Figure 20: the import mapping to correctly map the response from UbiOps 

Consequently, the last step in the microflow is to trigger a page to pop up with the value of the response (see figure 22). This means we need to create another page, with drag-and-drop elements, to which we pass the response value. 

Figure 21: show the result page (3) and pass the entity Photo

Step 4: Create a page to display the result 

As seen in figure 21, a page must be created that displays the variable “age” somewhere. By dragging and dropping the third page is created, and the selected data source (the variable “age”) is selected. 

Figure 22: Display of the response from the call to the UbiOps API. 

Step 5: Publishing the Mendix application

Last but not least, we will publish the application and ensure it’s accessible for the public by managing the permissions. See the mendix docs for more information on how to do this. 

You can try our app for yourself here:  https://ageestimationapp-sandbox.mxapps.io/

Wrapping up 

With two free-to-use platforms, an open-source AI model and limited knowledge of programming or software development, you can create your own AI-powered application. We did not focus on the performance of the deep learning model for this project, so you might find yourself to be estimated 20 years younger than you actually are. However, the aim is to illustrate the ease of building an end-to-end application with just two platforms and some knowledge of Python. 

Would you like to build something similar? Feel free to create a free account with Mendix and UbiOps. In case of any questions, let us know in the comments or simply reach out.  We hope you enjoyed this read and hopefully, we gave you some inspiration for your own project!