Using Google’s ML Kit to control an Android device without hands
The idea is to detect facial expression such as blinking one eye, to perform action(s) on an Android device.
In this article I will show how I created a simple application allowing the user to scroll forward by blinking with his left eye, and scrolling backward by blinking with his right eye. I will cover the main principle without entering in depth in the implementation to keep it simple.
Detect Facial expression
Through the Google ML Kit, we can use the Mobile Vision API to add Face Tracking to an application. Google provides a simple sample on GitHub. The given example is an app that uses the rear facing camera to show a view of the detected faces. The Face class has some build-in methods that we will use.
public float getIsLeftEyeOpenProbability()
public float getIsRightEyeOpenProbability()
public float getIsSmilingProbability()Based on the Google app sample, we create a class extending Tracker<Face>, and on the onUpdate method, we can simply define actions based on the characteristics of the face detected. We also change the camera used, from CAMERA_FACING_BACK to CAMERA_FACING_FRONT.
Simulate touch using an AccessibilityService
Android does not allow any third application to simulate touch screen. We need to use an AccessibilityService in order to get enough permission to perform scrolling for example.
When the service is created, and all permission are granted, we can create a GestureDescription.Builder to simulate a screen touch which can be started by calling displatchGesture.
Conclusion
I covered here the basic principle of how to use facial expression to control an Android device, using Google’s ML Kit. I only used build-in methods for this example, however, it is feasible to create more complex commands in order to get more control of the device, such as macro etc..
The full application is available on my GitHub. It has been made to manage permissions, overlay preview, and it allows the user to auto scroll.
The main purpose of this project was to be able to scroll on TikTok without using hands.
