Real-Time Image Classification On Android Using Flutter, TFlite

05 Nov, 2019 / Blog Posts By: Krishnaraj R

Making a smartphone app from scratch might seem a daunting task to most newbies and maybe even seasoned programmers.Enter Flutter.

As One of the promising, albeit untested technologies unveiled by Google for android app development, Flutter aims to make smartphone app development easier than ever. It is even rumored that flutter will soon be the go-to name in mobile app development.Creating UIs is a breeze and the developer friendly coding style makes the icing on the cake.

Working as an AI engineer ,one of the most common problems I and my team mates encounter is the matter of deployment of our models. How do you prototype your deep learning models in a quick,yet elegant way? The answer is carried around in our very pockets. Smartphones. That’s right. That overpowered piece of hardware on which you are possibly reading this article at this very moment is any developer’s dream come true. Combine the ubiquitous nature of smartphones and the versatile technology that is machine learning and you’re looking at a vast potential of innovation right there.
Now, I work at a place where people are often expected to roll up their sleeves and get the job done rather than wait for help. So, at one such instance, the matter of deployment loomed ahead of me and I had to choose between the daunting world of native android development and an exciting, yet uncharted new tech called Flutter. I chose the latter. At the time of writing this article, not every aspect of flutter is super stable, but it is certain that all of those problems shall disappear. In this article, I shall describe the important aspects of integrating deep learning models into an android app made with flutter. While I am no experienced android nor flutter developer, I was able to get a base version working in about a week. That’s how easy it is.

I assume the reader has a base understanding of layouts in flutter and a basic understanding of the programming language used,which is Dart. If you are unfamiliar with what I just mentioned a few beginner posts will get you on your way. While the methods and code mentioned in this project is not exactly what I would call optimised, it can serve as a starting point or a base for your own projects. So let’s dive in to it.

 

So let’s open up our pubspec.yaml file.

 

Here, you can see that I’ve specified the tflite: 1.0.1 as a dev_dependency. Now it is ready to be imported and used with our app. It’s that simple (Not quite). When flutter runs the packages get command, the files required to run this plugin will be downloaded without any further action from our side.
Now let’s open up our home.dart file, which is the page which has the ignite button labeled “start classification”.

 

importing the title plugin

 

As you can see, I’ve imported the tfliteplugin on line 3 in the home.dart page, similar to how I would import library files in other standard programming languages.
The Tflite plugin comes bundled with a function called loadModel() which is used to specify the name of the model to be loaded and the labels.txt file, which contains the name of the classes used.

 

Now let’s move on to the camera.dart file, where the actual inference takes place. Here, We use the function runModelOnFrame(), which takes the frame from the camera stream and takes it as input to the model. This function requires a list of arguments such as the frame, its height, width, the number of results to be displayed and so on.
The output is received in the format specified by the plugin’s ReadMe file found on Flutter Pub.