Evaluation of Cooking Recipes Using Their Texts and Images

摘要

Many people are using smartphone applications and websites with respect to cooking recipes in almost everyday life. By using cooking recipe sites, we easily search the cooking recipe for dishes that we want to eat and cook the dish based on the recipe. When we search cooking recipes, we refer to the two information that are pictures of cooked food and texts of cooking recipe. In this study, to investigate good cooking recipes, we built a multimodal coupled network to use these two kinds of information of cooking recipe sites. In this multimodal coupled network, as pre-learning, we learned images and texts separately. To learn texts, we first used word2vec to vectorize for each text, and then we use the deep averaging network to identify the objective function. To learn images, we performed transfer learning of images using VGG16 which is a famously trained model. We combined these two kinds of 300-dimensional feature vectors extracted from the final layer of the two models obtained by pre-learning, and then we learned a total of 600-dimensional feature vectors. As a result, the multimodal coupled network that combines image and text data has the best accuracy.