In order to reach the largest possible audience, and to leverage iOS’s recent ARKit SDK, we decided to build for the iPhone. Leveraging ARKit made the Augmented Reality portion of this project a snap to do.
For the object recognition functionality, we first experimented with Apple’s built-in ARKit Computer Vision features, but while they were fast, and we liked the ARKit’s augmented reality features, we believed that supplementing with TensorFlow would improve our accuracy rate, so we decided to build our own object recognition model. Using the TensorFlow machine learning framework, we trained the image recognition algorithm by feeding in tens of thousands of public domain photographs from Flickr with various food hashtags and labels. A web-based microservice (API) was built around the resulting computer vision algorithm. We deployed this to AWS, and wired up ARKit to use our new hand-rolled food-focused computer vision microservice.
The finishing touch was to implement a 3rd party API for nutritional information, and then display this information in Augmented Reality on the iPhone.
Foodalyzer is available for free in the App Store