Deepcartoon on UbiOps with AWS S3 and AWS Lambda

By 25 February 2021March 15th, 2021Blog

 

 

In this article, we are going to highlight some positive applications of generative adversarial networks (GANs) that we found on Github. We will demonstrate how to create a scalable GAN application for the creation of “Deep Cartoons” using UbiOps and AWS. During this process we will highlight some features of UbiOps (spoiler alert!). In the end we will test the operationalised GAN app on an image of our CEO, Yannick Maltha.

 

 

Awesome GAN

Deepfakes is definitely something that has received a lot of attention in the last year. One of the reasons for the widespread of deepfakes is the availability of new types of neural networks, namely GANs. These types of neural networks are controversial as there have been a large number of negative use cases. Many famous people have been subject to deep faked media. This is often referred to in discussions about Ethics and AI like in our recent user event https://ubiops.com/ubiops-user-event-ai-ethics-ai-audits-and-detecting-deep-fakes/.

Today we want to talk about the other side of the coin, because there are definitely a lot of positive applications. Let us highlight a couple of open source examples where GANs have been used in a positive way:

DeOldify (12.6K⭐) is a great open source Github project, where you can colorize and restore old and grayscale images (and videos!). It applies NoGAN for its training.

 

Source: https://github.com/jantic/DeOldify

StackGAN (1.7K ⭐) is an open-source project using GAN to generate images from captions. Would be really nice to create a movie from your old books with this technology!

 

Source: https://github.com/hanzhanggit/StackGAN

But today we would like to talk about another great project, and deploy it to UbiOps. Photo2cartoon (2.1K⭐) is a project that can generate a cartoon out of a picture in which it can detect a face.


Source: https://github.com/minivision-ai/photo2cartoon/blob/master/README_EN.md

 

How to make a user-facing app out of this?

As photo2cartoon is an algorithm and not yet a full fledged application we are going to do a couple of things. First, we will deploy photo2cartoon to UbiOps to operationalize the model. This will highlight some recent changes in the UbiOps backend.

Secondly, we will show how to build a user facing application out of the deployed algorithm using a web server, AWS S3, AWS Lambda and UbiOps.

The end-result of this project will be that any image uploaded to the web server will end up in S3 and trigger the photo2cartoon model to create a cartoon out of it. Finally, the application will be tested with a real picture of our CEO (CEOs like to be in the picture, right?).

 

Deploying the model on UbiOps and using Dockerization

The model that we would like to deploy in UbiOps (Photo2cartoon) has quite a lot of dependencies. For instance, it requires dlib for face recognition, which in turn requires cmake for compilation, and opencv is used in the background for many operations on the provided image. UbiOps allows you to package the model with all those dependencies. How does this work?

First of all, you configure UbiOps to create a new deployment for this model. Then you wrap the photo2cartoon algorithm in the UbiOps format using https://github.com/UbiOps/deployment-template. The resulting zip.file with the code and dependencies of photo2cartoon model can be uploaded to UbiOps (we call this a revision). This will create a docker image for you in the background in a building step. The docker image is deployed to our backend and an endpoint is created. Finally, a validation step is done to ensure the model does not have any initialization errors.

However, if you do not upload new code, but only change some parameters (e.g. allocated memory), then only the deploying step is executed.

 

UbiOps governs your algorithm by keeping track of the revisions. In this way you know exactly which code is running and the deployed code is immutable. You can also download back each uploaded revision. In the near future we plan to support reverting to a previous revision and to allow us to implement fully customized images. Please also take a look on other advantages of using UbiOps https://ubiops.com/product/

Compared to our previous approach where we loaded the entire model code into memory this has some additional advantages:

 

  • This method supports including linux libraries instead of only python packages.
  • If the docker image is already available on the machine, cold start time is lower. If not, only the additional layers containing the model code and dependencies need to be downloaded. The base image is prepulled.
  • Now only a Docker image needs to be downloaded. Previously a Docker image and a the machine learning model;
  • Images are immutable. This means if we change our runtime the old images will still work.

 

Scalable image transformation app

Just hosting a machine learning model is not enough. We need an easy to use, scalable, and robust image transformation app. Our envisioned end-users are definitely not getting direct access to the hosted algorithm. Instead a web service is needed to hide the details of our algorithm and backend from the end user and provide decoupling between the AI layer and the user facing application. The user facing application will include a user friendly website, authentication, rate limiting, maximum image size, image format check, DNS, TLS, etc.

The web service will store the uploaded images in S3 or similar storage. S3 has a couple of nice features that we would like to use. E.g., it can trigger an AWS Lambda function which calls photo2cartoon once the image is uploaded to S3. Another benefit is that in S3 we can set image retention. Since many of the images will contain a human face, we can delete the images as soon as they are not needed anymore for the functioning of the application.

 

See the picture above for a high level architecture of what has been discussed previously. In this way we have a scalable application for image transformation. It also allows us to replace the components of our stack individually, e.g. by upgrading the photo2cartoon model or our webservice, or to add other types of image translation models.

 

The end-result of our CEO

Now we have hosted photo2cartoon in UbiOps and provided a reference architecture for a user-facing application, we are ready to test it out!

See the two images below. The cartoon is quite ‘neutral’ by itself unlike some deepfakes. The cartoon looks a lot like our CEO, but there are some visible errors around the nose and the right eye. We definitely see some room for improvement for the next version of our algorithm.

Credits

 

Yannick Maltha (CEO of UbiOps) for sharing his profile picture