Foodalyzer is a quick and easy way to get nutritional information about your meal – just point your phone’s camera at your food and nutritional labels will appear in an augmented reality environment.
Foodalyzer was designed as a proof-of-concept that food items on store shelves could be recognized using AI and Computer Vision. The Computer Vision results needed to be cross-referenced against a database of food items to confirm that they were indeed food. After building this proof-of-concept, the product team decided that the app had real-world value of its own–or would, if we could provide nutritional information about the foods identified on-screen. We discovered that we could do this using publicly available nutritional facts and Foodalyzer was born. The larger project effort is still ongoing.
In order to reach the largest possible audience, and to leverage iOS’s recent ARKit SDK, we decided to build for the iPhone. Leveraging ARKit made the Augmented Reality portion of this project a snap to do.
For the object recognition functionality, we first experimented with Apple’s built-in ARKit Computer Vision features, but while they were fast, and we liked the ARKit’s augmented reality features, we believed that supplementing with TensorFlow would improve our accuracy rate, so we decided to build our own object recognition model. Using the TensorFlow machine learning framework, we trained the image recognition algorithm by feeding in tens of thousands of public domain photographs from Flickr with various food hashtags and labels. A web-based microservice (API) was built around the resulting computer vision algorithm. We deployed this to AWS, and wired up ARKit to use our new hand-rolled food-focused computer vision microservice.
The finishing touch was to implement a 3rd party API for nutritional information, and then display this information in Augmented Reality on the iPhone.