At doc.ai one of our goals is to bring the power of machine learning to the devices we use every day. In a recent post on the medical selfie, our Co-Founder and COO, Sam De Brouwer, wrote about the advantages of leveraging machine intelligence on edge devices such as our mobile phones, including speed, privacy, and personal control. In that post she discussed doc.ai’s new mobile machine learning model that, from a selfie which never leaves your phone, can infer biometric data such as your age, sex, height, and weight.
As part of our effort to accelerate the pace of mobile machine learning doc.ai is excited to announce two open source projects, Net Runner and TensorIO. Both are available on GitHub and both have been published with a permissive Apache 2 open source license. We’re using these tools right now at doc.ai as part of our medical selfie work, and we can’t wait to see what direction the community takes them in.
Net Runner is doc.ai’s on device environment for prototyping and evaluating computer vision machine learning models. With Net Runner, data scientists who otherwise have no mobile software experience can quickly debug and test their computer vision models on iPhones.
Net Runner ships with a number of MobileNet image classification models that are a lot of fun to use. The models run in real time on input coming from either of the phone’s cameras and can identify objects in a thousand classes. Point the phone at the world and the phone tells you what it sees.
With a little knowledge of TensorFlow and JSON, data scientists can add their own models to the application and see the output in real time. Models can be measured for latency and accuracy, run on hundreds or thousands of still images in iPhoto albums, and automatically tested using a headless evaluation mode.
We built Net Runner specifically to help us develop and iterate on the biometric selfie. We have a number of other models in the pipeline that Net Runner will be helping us with, and we’re excited to see how other researchers use it. Make requests and contribute to the app over at our Net Runner GitHub repository, or download the app from the iPhone App Store. We’d love to know what you think!
TensorIO is a portable framework for iOS that removes the need for direct interaction with the TensorFlow Lite library and replaces it with a convenient JSON interface. The library was developed alongside Net Runner and is the foundation on which Net Runner is built. It is the underlying code which makes it easy for data scientists to run a wide variety of models on mobile phones without the need for mobile development experience.
And we mean it. We built TensorIO to be accessible. Without knowledge of C/C++, Objective-C, or Swift, model builders are able to describe the inputs and outputs to their models in JSON and can then run those models in Net Runner or any other application that ships with the TensorIO library. Of course, for teams with mobile experience, TensorIO is customizable and extensible, and our embedded documentation should help you get started.
TensorIO is one of the fundamental pieces of infrastructure with which doc.ai is bringing machine learning to edge devices such as our mobile phones. We have a lot in store for this project and are looking forward to your feedback and contributions. Find the source code and extensive documentation both for data scientists and developers at the TensorIO GitHub repository.
At doc.ai we believe in the power of edge computing and machine learning on device, and we recognize the importance of data privacy and personal control over what information you share. With these two projects we aim to advance our mission of decentralizing medicine and transforming healthcare with machine intelligence. We hope you’ll join us.