When eating out, many people photograph their food before digging in. What if those photos could help you recreate the dish at home?
It sounds like the fictional failed app SeeFood from the HBO show Silicon Valley but it’s real (and more effective) thanks to researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL).
The researchers have developed an artificial intelligence system they’ve dubbed “Pic2Recipe” that will provide a list of ingredients and even an actual recipe just from a photo of your meal.
The idea seems simple — upload a photo and in return, the system predicts the ingredients and taps into a database to suggest several recipes it determines to be similar — but the execution has been the result of some hard work.
“Understanding food images is a difficult problem that even research teams at places like Google have trouble with,” said Nick Hynes, an MIT CSAIL graduate student and lead author of the paper that details the system.
“Researchers have explored this before, but have used much smaller datasets or ones that don’t generalize as well as to different types of cuisine,” he said. “Recently, Pinterest developed an algorithm for food recognition, but is simply recognizes the type of food — i.e. ‘turkey,’ ‘banana’ — rather than drawing on actual recipes.”
The researchers have been working on their system for about a year, Hynes explained, during which they’ve built a database of more than 1 million recipes and annotated those with more information about the ingredients.
Pic2Recipe doesn’t only identify ingredients based on image, but has also been trained to find patterns, make connections and to consider the meal as a whole.
“Much like a human observer, it looks at the context of the dish,” Hayes said. “For example, capers and peas look similar, but from context it might find capers on a bagel and peas served with a side of potatoes. Even if it can’t identify a few ingredients in the dish, the broad classes of foods are likely distinct enough for it to retrieve something reasonable.”
Researchers see people using this to track nutrition or take a picture of a meal at a restaurant and know what goes in it, so they can try their hand at home.
The system is still in the early stages, however, and Hayes said it needs some improvement identifying blended foods like smoothies and soups, or distinguishing between similar dishes like a BLT versus a reuben.
Still, in experiments conducted so far, the system found the right recipe 65 percent of the time. It’s particularly good at identifying cookies and muffins, which are a main theme of the database, Hayes said. After seeing a picture of sugar cookies, the system identified eight out of 11 ingredients.
“We took photos from sites like AllRecipes and Food.com, with no restrictions on the type of food,” he said. “That said, the recipe database is definitely heavy on desserts, baked goods and cocktails. After all, what do people like more than drinks and sweets. The model very much reflects this bias.”
