JP2019023850A - Information processing system, information processing method and information processing program - Google Patents

Abstract

To provide an information processing system, an information processing method and an information processing program capable of accurately detecting a cuisine section in an image.SOLUTION: An information processing device comprises: a storage section which stores utensil data representing a shape, a pattern, a color, or texture of a dish or the combination thereof and correspondence data representing correspondence between the dish and a cuisine using the same; an image acquisition section which acquires an image; a utensil detection section which determines the dish included in the image on the basis of the acquired image and the stored utensil data; and an image extraction section which extracts an image region of the cuisine corresponding to the determined dish from the image on the basis of the correspondence data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing system, an information processing method, and an information processing program.

  2. Description of the Related Art Conventionally, there is a technique in which a feature image of a dish is extracted from image data that has been imaged and input, compared with a feature image registered in a storage means, and a corresponding food type is detected and determined. In addition, a technology that extracts each tableware on the tray in an image obtained by picking up a food bowl on which food is placed on each tableware on the tray and discriminates the type of tableware and the position on the tray in light of a tableware database input in advance There is. There is also a technique for determining a meal portion included in an image. In addition, there is a technique for generating a dish image by cutting out each dish area in a photographed image captured using a dish as a subject. In addition, there is a technique for detecting a subject area of an image and generating a feature amount including position information of the detected subject area (see, for example, Patent Documents 1 to 5).

  In the conventional technology, for example, feature data indicating the color or shape of an object is prepared in advance, and the object included in the photographed image is recognized by comparing the photographed image with the feature data. Can be extracted.

JP 2003-173369 A Japanese Patent Laying-Open No. 2005-070908 JP 2012-212249 A JP2013-255728A JP 2013-045441 A

  However, there are various dishes in various genres. Even in the same dish, cooking methods such as utensils and cooking procedures, ingredients to be used, and arrangement methods may differ. If the cooking method is different, the characteristic data such as the color and shape included in the photographed image of the dish may differ greatly even in the same dish. Therefore, when a dish is detected in comparison with feature data, for example, a dish portion cannot be detected even if the photographed image is included in the photographed image, or a part other than the dish in the photographed image is cooked. The detection accuracy of the cooking part included in the photographed image may be reduced due to an erroneous detection that detects it as a part or a dish part even though the dish does not exist in the photographed image. there were.

  On the other hand, in order to improve the extraction accuracy of the cooking portion for various dishes, it is necessary to prepare a huge amount of feature data such as cooking methods in advance for each dish, increasing the capacity load of the database In addition, the data processing load may be high.

  The present invention has been made in view of the above points, and provides an information processing system, an information processing method, and an information processing program capable of accurately and easily extracting a cooking portion included in a captured image. One of the purposes.

  In order to solve the above-described problem, in one embodiment of the present invention, the information processing system stores appliance data indicating the appearance of an appliance that arranges dishes, and tableware correspondence data indicating correspondence between the appliances and the dishes. An image acquisition unit that acquires an image, a detection unit that detects an appliance that is the same as or similar to the appearance indicated by the appliance data from the acquired image, and the tableware correspondence data of the appliance that is detected An image extraction unit that determines the determined dish and extracts an image area of the determined dish from the image.

  In order to solve the above problem, in one embodiment of the present invention, an information processing method stores appliance data indicating an appearance of an appliance that distributes dishes and tableware correspondence data indicating correspondence between the appliances and the dishes. A storage step, an image acquisition step of acquiring an image, a detection step of detecting an appliance having the same or similar appearance as the appliance data from the acquired image, and the tableware correspondence data of the detected appliance Determining an associated dish and extracting an image area of the determined dish from the image.

  In order to solve the above-described problem, in one embodiment of the present invention, the information processing program stores appliance data indicating the appearance of an appliance that distributes dishes, and tableware correspondence data indicating correspondence between the appliances and the dishes. Storage processing, image acquisition processing for acquiring an image, detection processing for detecting an appliance that is the same as or similar to the appearance indicated by the appliance data from the acquired image, and the tableware correspondence data of the detected appliance. The associated dishes are determined, and an image extraction process for extracting an image area of the determined dishes from the images is executed by a computer.

  According to one embodiment of the present invention, it is possible to provide an information processing system, an information processing method, and an information processing program that can accurately extract a cooking portion included in an image.

It is a block diagram which shows an example of a function structure of the information processing apparatus in embodiment. It is a block diagram which shows an example of the hardware constitutions of the information processing system containing the information processing apparatus in embodiment. It is a figure which shows an example of the appliance data utilized with the information processing apparatus in embodiment, and corresponding | compatible data. It is a figure which shows an example of the dish data utilized with the information processing apparatus in embodiment. It is a flowchart which shows an example of operation | movement of the information processing apparatus in embodiment. It is a flowchart which shows an example of the tableware determination process of the information processing apparatus in embodiment. It is a figure which shows the 1st example of the image area extraction of the dish performed with the information processing apparatus in embodiment. It is a figure which shows the 2nd example of the image area extraction of the dish performed with the information processing apparatus in embodiment.

  Hereinafter, an information processing system, an information processing method, and an information processing program according to an embodiment of the present invention will be described in detail with reference to the drawings.

[Functional configuration of information processing device]
FIG. 1 is a block diagram illustrating an example of a functional configuration of the information processing apparatus according to the embodiment.

  In FIG. 1, the information processing apparatus 10 has functions of an image acquisition unit 101, a detection unit 102, an image extraction unit 103, a dish analysis unit 104, and a storage unit 105.

  The above-described functions of the information processing apparatus 10 in the present embodiment will be described as functions modules realized by, for example, an information processing program (software) that controls the information processing apparatus 10.

  The image acquisition unit 101 acquires an image. The image acquired by the image acquisition unit 101 includes, for example, a captured image obtained by shooting a dish. “Cooking” refers to food that can be eaten in a meal. In addition to what cooked ingredients, the dish in this embodiment may contain the ingredients etc. which can be eaten as it is, without cooking. For example, vegetables and fruits are cooking materials, and when they are put on the table as they are, they become dishes in this embodiment. The dish may contain a liquid such as juice or soup.

  The captured image includes at least one of a still image and a moving image. The image acquisition unit 101 may acquire an image stored in the information processing apparatus 10 or may acquire an image from outside the information processing apparatus 10. The outside of the information processing apparatus 10 is, for example, a client apparatus connected via a network described later. The information processing apparatus 10 extracts an image area of a cooking portion from the image acquired by the image acquisition unit 101. However, the image acquired by the image acquisition unit 101 may include an image in which a dish is not photographed, such as a simple landscape photograph. The image acquisition unit 101 may temporarily store the acquired image in the storage unit 105 or the like.

  The storage unit 105 includes appliance data, dish arrangement data, tableware correspondence data, cutlery correspondence data, and dish data. The appliance data is data indicating the appearance of tableware, cooking utensils, or cutlery serving food. The appearance is, for example, the shape, pattern, color, or texture of the device, or a combination thereof. The dish arrangement data is data indicating the correspondence (corresponding arrangement) between the appearance of dishes and the like and the arrangement of dishes when dishes are arranged on tableware or cooking utensils. The tableware correspondence data is data of dishes corresponding to each tableware and the like that are highly related to the tableware and the like. Further, the cutlery-corresponding data is dish data corresponding to each cutlery that is highly related to the cutlery. The dish data is data indicating the feature amount of the dish.

  The storage unit 105 may store these data for a long time like a database, or may temporarily store data acquired from an external device such as a cache memory.

<Equipment data>
The appliance data is data of tableware and cooking utensils for serving dishes, or cutlery. Dishes are arranged on tableware and utensils, and meals are made at cutlery. Therefore, when photographing a dish, there is a high possibility that an instrument is included in the photographed image along with the dish. Tableware and cooking utensils in the utensil data include the following items.
椀: Rice bowl, soup bowl, donburi Plates and plates: Western dishes, Japanese dishes, glass dishes, Western dishes, Japanese dishes Plates and bowls: Large bowls, small bowls, salad bowls, dessert bowls Cups: mugs, teas Cup Glass / Sakeware: Tumbler, Wine Glass, Mouth, Beer Glass Cookware: Frying pan, pan, kettle, hot plate, kitchen appliances The tableware and cooking utensils above are for the purpose of arranging solid or liquid dishes in the meal. Since it is used, it may be called a “container”. Since the pan and the pan may also be placed on the table with the cooked food as it is, in the present embodiment, it is converted into data in the same instrument data as that of the plate or the like. In the present embodiment, it is not necessary to strictly distinguish tableware from cooking utensils in that dishes are arranged. Accordingly, in the following description, “tableware” includes cooking utensils.

Moreover, the tableware in appliance data includes the following cutlery data, for example. Chopsticks: chopsticks, disposable chopsticks, iron chopsticks Cutlery: spoons, forks, knives, lotus root, crab spoons, ladle, iron In this embodiment, chopsticks, straws, toothpicks, skewers, etc. are also “cutlery”.

  The shape of the appliance is, for example, the shape and size of tableware. For example, there are various types of dishes such as round plates, square plates (square, rectangular), bean plates, small plates, large plates, octagonal plates, and the like. The pattern of the instrument is a decoration applied to the surface of the instrument, such as a plain pattern, a flower pattern, a Chinese pattern (dragon, dragon, thunder, and so on). The color of the device is a color of the surface of the device and includes a single color or a plurality of colors.

  The texture of the device is the texture of the surface of the device, and is the way the hair tips such as the appearance gloss and hairline are aligned, or the touch (hand) when directly touching with the hand. In the present embodiment, the texture may be expressed as a texture. For example, the texture of a pottery container in Japanese tableware has a rough texture with fine irregularities on the surface. On the other hand, the texture of the porcelain container in Western tableware is smooth and glossy.

  The appliance data is data indicating the shape, pattern, color or texture of the appliance or a combination thereof, and can be quantified as a feature amount in each appliance. The feature amount is a numerical value of the shape, pattern, color, or texture of the appliance, for example, in machine learning, and can be expressed as a feature vector. As machine learning, for example, an algorithm such as a neural network or a support vector machine can be used. The shape, pattern, color, or texture of the appliance does not exist alone, but they combine to form the appearance of the appliance. In machine learning, a plurality of captured images are learned for one type of appliance, and the shape, pattern, color, or texture of the appliance is learned. The dishes used (uses) vary depending on the shape and the like. In the present embodiment, it is possible to learn the relationship between an appliance and an application by learning one appliance for each application. The storage unit 105 stores in advance appliance data digitized by learning. The memory | storage part 105 may memorize | store apparatus data about the tableware which actually exists, for example in a house etc. and a dish may be arranged.

  The correspondence data stored in the storage unit 105 is data indicating the correspondence between the above-described appliance or cutlery and a dish using the appliance or cutlery. Hereinafter, dish arrangement data, tableware correspondence data, and cutlery correspondence data will be described as correspondence data.

<Cooking arrangement data>
Each tableware used as a container has a shape, and it is possible to correspond in advance to which position of the tableware the dish is easily arranged. For example, in a deep round dish, there is a high possibility that a dish is arranged in the center of the round dish. On the other hand, there is a low possibility that dishes are arranged only on the outer edge of the round plate. In addition, there is a high possibility that dishes will be arranged on the rectangular Japanese food dishes so that they stick out to the outside of the dish like Sanma's salt grill, or the dishes are distributed in the longitudinal direction of the dishes. On the other hand, it is unlikely that food will be served only on a portion of the Japanese dishes. That is, it is possible to predict in which position of the container the dish is arranged in correspondence with the shape of each container. The dish arrangement data is data for predicting the position at which the dish is arranged for each container, and is stored in the storage unit 105 to predict the dish image area and extract the dish image area in pattern matching. The load can be reduced.

<Tableware data, cutlery data>
There is a close combination of tableware and cooking. For example, Japanese dishes are used for Japanese dishes, Western dishes are used for Western dishes, Chinese dishes are used for Chinese dishes, and Korean dishes are easily used for Korean dishes. In other words, depending on the type of tableware used for cooking, the type of cooking may be predicted to some extent. For example, a dish arranged in a container having a lightning pattern is likely to be a Chinese dish. Moreover, the dish arranged on the iron plate is likely to be a meat dish. In addition, as described above, the dishes arranged in the rectangular Japanese tableware are likely to be grilled fish and Japanese food.

  Similarly, cutlery has a combination closely related to a predetermined dish. For example, iron chopsticks are often used for Korean food, and lotus root is often used for Chinese food such as ramen and fried rice. Chopsticks are used in Japanese food, and forks and knives are used in Western food. In other words, the type of dish can be determined to some extent by the cutlery. Cooking may be further determined, for example, by the combination of containers and cutlery. For example, when a lotus is combined with a dish arranged in a container with the above-mentioned lightning pattern, the dish is likely to be fried rice, and when the dish is combined with chopsticks, the dish is ramen Etc. are likely. In other words, it may be possible to predict what kind of dish is arranged for each tableware (corresponding dish). If the dish can be predicted, it is possible to predict the arrangement position of the dish and the feature amount of the photographed image of the dish. The storage unit 105 stores the corresponding dish for each container, thereby predicting the dish image area and reducing the load of extracting the dish image area in pattern matching. Tableware correspondence data and cutlery correspondence data may be collectively referred to as correspondence data.

  The relationship between food and tableware is greatly influenced by food culture and lifestyle. For example, in a Japanese dining table, chopsticks may be used for meals even for Western dishes, and the relationship between chopsticks and Japanese food is not strong. On the other hand, knives and forks are unlikely to be used for Japanese dishes at Japanese tables, and the relationship between knives, forks and Western food is strong. Therefore, it is desirable to prepare the correspondence data stored in the storage unit 105 according to, for example, the country, the region, and the age. For example, the storage unit 105 may have correspondence data for East Japan and correspondence data for West Japan individually.

  Moreover, you may make it produce corresponding data based on the picked-up image which image | photographed the cooking | publishing published on the cooking recipe site. For example, in a cooking recipe site, a plurality of shot images of cooking examples may be posted and posted for one dish (recipe). Corresponding data can be easily created by analyzing in advance the tableware used in each posted photographed image.

<Cooking data>
The dish data is data (feature data) indicating the feature amount of the dish. The feature amount of a dish is a feature amount of an image in a photographed cooking portion (area) when the dish is photographed. The feature amount of the dish includes a plurality of feature amounts such as a feature amount indicating the color feature of the dish and a feature amount indicating the feature of the dish shape. The feature amount of cooking can be quantified by machine learning as described above, for example, an image of a cooking portion included in the image. In machine learning, which feature quantity among a plurality of feature quantities determines as a dish may differ depending on the depth of learning. For example, in some dishes, the determination accuracy based on the shape of the dish may be high, and in other dishes, the determination accuracy based on the color of the dish may be high. In machine learning, the determination algorithm can be corrected by itself according to the number of samples to be learned.

  The dish data used for machine learning can be acquired by learning a photographed image obtained by photographing a dish posted on a dish recipe site, for example. In the cooking recipe site, taken images of food are posted from many users, and the number of recipes may reach several million. For example, it is possible to easily digitize the feature amount of a dish in the recipe by machine learning of a shot image of the dish cooked based on the same recipe. For example, a post image and a report image, which will be described later, are cooked based on the same recipe information, and image data to be machine-learned can be efficiently collected to improve detection accuracy. In addition, while cooking images may differ due to slight differences (variations) in recipes and cooking methods, when a photograph is viewed by a person, it may be photographed even if there are some color differences or shape differences. In some cases, it is easy to distinguish the dishes. By making the machine learning the feature amount of the recipe that is posted on the cooking recipe site, it is possible to store the cooking data corresponding to the cooking variation in the storage unit 105.

  The detection unit 102 detects an appliance that is the same as or similar to the appearance indicated by the appliance data from the image acquired by the image acquisition unit 101. The detection unit 102 further detects a cutlery that is the same as or similar to the appearance indicated by the cutlery data from the image acquired by the image acquisition unit 101. Further, the detection unit 102 detects an appliance based on the texture data stored in the storage unit 105.

  For example, the detection unit 102 detects tableware by comparing at least one feature amount of shape, pattern, color, or texture included in the appliance data with a feature amount included in the acquired image. For example, the detection unit 102 detects tableware based on whether or not the feature amounts included in the appliance data match or are similar within a predetermined range. The detection unit 102 may detect the tableware having the closest feature amount from the appliance data. The detection unit 102 may determine the priority order among the feature quantities of the shape, pattern, color, or texture, and use the feature quantity having a higher priority as a condition for determining tableware.

  In addition, the detection unit 102 determines the cutlery included in the captured image. The detecting unit 102 estimates the type of dish based on the determined cutlery and the corresponding dish data included in the appliance data. For example, when the captured image includes a large spoon, the detection unit 102 estimates a corresponding dish that is highly likely to be used.

  When the captured image is a moving image, a still image can be acquired by capturing the moving image at a predetermined timing. By performing image processing such as tableware determination on a still image, the processing load can be reduced.

  In addition, you may enable it to set beforehand the similar range of the external appearance of an instrument or a cutlery. For example, the use (cooking) of a spoon may vary depending on the size. On the other hand, even if the dish is the same dish, the size and shape may differ depending on the number of people. By setting the similar range in advance, it is possible to detect different appliances as different appliances and improve the determination accuracy of dishes in the image extraction unit 103.

  The image extraction unit 103 determines a dish associated with the tableware correspondence data detected by the detection unit 102 based on the tableware correspondence data stored in the storage unit 105, and obtains an image of the determined dish image area. Extracted from the image acquired by the unit 101. The image extraction unit 103 further determines a dish associated with the cutlery correspondence data detected by the detection unit 102 based on the cutlery correspondence data stored in the storage unit 105, and the image of the decided dish A region is extracted from the image acquired by the image acquisition unit 101.

  The image extraction unit 103 sequentially reads the feature amount of the dish included in the dish data, and checks the presence or absence of an image whose feature amount is identical or similar within a predetermined range in each dish. The image extraction unit 103 can detect a cooking portion and other portions (tableware portion, background portion, etc.) from the feature amount of the image, and extract the continuously detected cooking portion as a cooking image area.

  The image extraction unit 103 preferentially confirms the feature amount for the type of dish determined by the detection unit 102. As a result, for example, the amount of data to be compared is reduced compared to the case where the features included in the dish data are read out at random, and the image regions having the same or similar features are easily determined. It becomes possible. As a result, the extraction load of the image area of the cooking portion can be reduced, and the extraction (detection) speed and accuracy of the image area of the cooking portion can be improved. If the image extraction unit 103 cannot detect an image with an approximate feature amount, the image extraction unit 103 determines that no dish is included in the captured image. The image extraction unit 103 may provide an image of the extracted dish image area, for example, outside the information processing apparatus 10.

  Further, as described above, the corresponding arrangement indicates in which position of each container the food is likely to be arranged. The image extraction unit 103 predicts an image area of the dish from the container image based on the determined corresponding arrangement of the containers. The image extraction unit 103 extracts a dish image region from the predicted image region. For example, the image area of the dish is extracted by confirming whether or not the feature amount of the dish included in the dish data stored in the storage unit 105 matches or is similar to the predicted image area. Can be executed.

  In the present embodiment, by checking the feature amount of the dish based on the corresponding arrangement, it is possible to reduce the load of extracting the image area of the dish portion compared to the case of checking the feature amount of the dish for the entire captured image. It becomes. In addition, by not confirming the portion where the possibility of food being arranged is low, the extraction (detection) accuracy of the image area of the food portion is increased, and the possibility of erroneous extraction of portions other than the food is reduced. be able to.

  The dish analysis unit 104 performs an analysis process on the image area of the dish portion extracted by the image extraction unit 103. For example, the dish analysis unit 104 analyzes the amount of dishes (for how many people) from the size of the extracted image area. In addition, the dish analysis unit 104 may determine main dishes and side dishes from the positional relationship between the extracted image regions.

  Note that the functions of the image acquisition unit 101, the detection unit 102, the image extraction unit 103, the dish analysis unit 104, and the storage unit 105 included in the information processing apparatus 10 have been described as being realized by software as described above. However, at least one of the functions of the information processing apparatus 10 may be realized by hardware.

  In addition, any one of the functions that the information processing apparatus 10 has may be implemented by dividing one function into a plurality of functions. In addition, the information processing apparatus 10 may implement any two or more functions in a single function.

  Further, the information processing apparatus 10 may be an apparatus realized by a single casing or a system realized by a plurality of apparatuses connected via a network or the like. That is, the information processing apparatus 10 is a form of the information processing system in the present embodiment. Further, the information processing apparatus 10 may be a virtual apparatus such as a cloud service provided by a cloud computing system. In addition, the information processing apparatus 10 may be a general-purpose computer such as a server apparatus or a dedicated apparatus having a limited function.

  Further, at least one of the above functions of the information processing apparatus 10 may be realized in another apparatus. That is, the information processing apparatus 10 does not have to have all the functions described above, and may have some functions.

[Hardware configuration of information processing system]
FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing system including the information processing apparatus 10 according to the embodiment.

  The information processing system 1 includes an information processing device 10, a display device 21, an input device 22, a client device 3 (3A, 3B, 3C, etc.), a server 4, a network 91, and a network 92.

  The information processing apparatus 10 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, a HDD (hard disk drive) 14, a display control unit 15, an input control unit 16, and a communication. An I / F (Interface) 17 is included.

  The information processing apparatus 10 is a general-purpose apparatus such as a server apparatus, a desktop PC, or a tablet PC, or an apparatus dedicated to information processing. The information processing apparatus 10 executes the information processing program described with reference to FIG.

  The CPU 11 controls the information processing apparatus 10 by executing an information processing program stored in the RAM 12, ROM 13, or HDD 14. The information processing program is acquired from, for example, a recording medium on which the information processing program is recorded, a program distribution server via a network, or the like, installed in the HDD 14, and stored in the RAM 12 so as to be readable from the CPU 11. The RAM 12, ROM 13, or HDD 14 may store data processed or processed by the CPU 11 in addition to programs executed by the CPU 11. For example, the storage unit 105 described in FIG. 1 can be realized by the RAM 12, the ROM 13, or the HDD 14. Note that the CPU 11 may be composed of a single arithmetic device or a plurality of arithmetic devices. The CPU 11 includes an arithmetic device such as a GPU (Graphics Processing Unit). The GPU is an arithmetic device dedicated to image processing, and is hardware effective in image processing such as determination of the type of dish or extraction of feature values of dish in the present embodiment.

  The display control unit 15 controls display data displayed on the display device 21. The input control unit 16 controls input data input from the input device 22. The display device 21 and the input device 22 may be, for example, a touch panel having an operation input function and a display function.

  The communication I / F 17 controls communication with a plurality of client devices 3 connected via the network 91. The communication I / F 17 controls communication with the server 4 connected via the network 92.

  The client device 3 is, for example, a PC or a smartphone that has a recipe posting function and a report posting function and transmits a captured image in which a dish is captured to the information processing apparatus 10. The client device 3 includes an application program that executes a recipe posting function and a report posting function and a Web browser, while the information processing device 10 provides a recipe posting service and a report posting service to the client device 3. It may be.

  Here, the recipe posting function is a function for posting a recipe created by the user. In the recipe posting function, it is possible to post a cooking title, ingredients and the amount thereof, a cooking method, a cooking image of the cooking result, and the like. In addition, the report posting function is a function in which a recipe posted by the recipe posting function is referred to by another user and a photographed image of the cooking result cooked based on the recipe is posted. That is, the post image of the food posted by the recipe posting function and the report image of the food posted by the report posting function are the same food cooked by the same recipe, and there are a plurality of images of the same food cooked by different users. Will be posted from the client device 3.

  The posted image and the report image in the present embodiment are suitable as machine learning for extracting feature amounts in the above-described appliance data and cooking data. For example, when trying to collect images of dishes to be used for machine learning by searching the Internet using the names of dishes, etc., images of dishes with completely different appearances such as ingredients and cooking methods, and what to collect There are cases where many noises such as images of completely different dishes are hit and it takes time and effort to select images that can be used for machine learning. Since the posted image and the report image are cooked based on the same recipe information as described above, the above-described noise is not mixed, and time and labor for image selection are not required. In addition, there are many differences in feature amounts between the posted image and the report image in a range in which a person can recognize that they are the same dish, and the data is suitable for improving the recognition accuracy of the same dish. In addition, since the posted image and the report image are posted by many users, it is possible to efficiently collect the image data to be learned.

  For example, the server 4 is a server having a data supply function for providing appliance data and cooking data to the information processing apparatus 10. For example, the data stored in the storage unit 105 described in FIG. The stored data may be indirectly indicated. Further, the server 4 may be a server that records the result of the image area of the dish extracted by the information processing apparatus 10.

[Data structure of instrument data and corresponding data]
FIG. 3 is a diagram illustrating an example of appliance data and correspondence data used in the information processing apparatus 10 according to the embodiment.

  In FIG. 3, the appliance data includes data items “tableware NO.”, “Major classification”, “small classification”, “shape feature value”, “pattern feature value”, “color feature value”, and “texture feature value”. Have The correspondence data includes data items of “corresponding arrangement” and “corresponding dish”. FIG. 3 shows a case where the instrument data and the correspondence data are integrated in the same table. However, the instrument data and the corresponding data may be configured in different tables. Moreover, although the case where the instrument data and the corresponding data are configured in a table format is shown in FIG. 3, the instrument data or the corresponding data may be configured in, for example, an XML (Extensible Markup Language) data format.

  “Tableware No.” is a data item to which a number that uniquely defines tableware or cutlery registered in the appliance data is input, and a different number is given depending on the shape of the tableware or the like. The “major category” and “minor category” are data items into which tableware categories are input, and containers and cutlery are classified into respective categories.

  The “shape feature value”, “pattern feature value”, “color feature value”, and “texture feature value” are data items that represent the feature value of a captured image of a tableware or the like. These feature amounts may be recorded at least within a range in which appearance can be taken. These feature amounts are compared in a captured image in a state where dishes are arranged. Therefore, it is not necessary to record the feature quantities that are not included in the captured image, such as the back and bottom surfaces of the container. In addition, when shooting a dish, for example, a dish is often shot obliquely from above. For this reason, in these data items, it is desirable to record the characteristic amount when the tableware is obliquely viewed. For example, in a bowl, the top and outside of the bowl are often photographed, and the inside bottom of the bowl is rarely photographed. Therefore, in the feature value, priority is given to the recording of the feature value of the inner upper part and the outer feature over the feature value of the inner bottom part.

  When determining tableware, etc., use any feature quantity of “shape feature quantity”, “pattern feature quantity”, “color feature quantity”, and “texture feature quantity” in any priority or combination. Whether to do this may be set according to the learning result in machine learning, for example. For example, in the present embodiment, the same tableware NO. Even in tableware to be categorized as, there may be a difference in the feature value depending on the use date and production lot. These differences can be recognized as the same tableware by causing machine learning of the features of the captured images of the plurality of tableware.

  The “corresponding arrangement” is a data item for recording dish arrangement data for predicting a position where a dish is arranged for each container. In the corresponding arrangement, a predetermined area is designated with respect to the shape of the container. For example, if the container is taken from an oblique angle, the corresponding arrangement is also inclined at that angle. For example, the corresponding arrangement of the donburi is a circular range along the donburi ridge when the donburi is photographed from directly above. However, if the image is taken from an oblique angle, a part of the corresponding arrangement of the donburi becomes a shadow of the donburi depending on the angle.

  The “corresponding dish” is a data item for recording dish correspondence data and cutlery correspondence data indicating a dish or a kind of dish (category etc.) set for one tableware or the like. In the “corresponding dish”, a plurality of kinds of dishes can be recorded. In addition, a priority order may be set for a plurality of types of dishes recorded in “corresponding dishes” such as one tableware. In addition, when it is difficult to estimate a dish due to tableware or cutlery, recording in a data item may be omitted. In addition, since the relationship between dishes etc. and the dish which uses the dishes etc. has a large regional difference etc. as mentioned above, you may make it prepare corresponding data for each area | region etc. in a corresponding dish. Corresponding dishes may be classified and registered in a plurality of layers and tags. For example, classification such as “Western food”, “Japanese food”, “rice”, “soup”, “breakfast”, and “lunch” may be registered in the corresponding dish.

[Data structure of dish data]
FIG. 4 is a diagram illustrating an example of dish data used in the information processing apparatus 10 according to the embodiment. For example, the dish data is stored in the storage unit 105 described with reference to FIG.

  In FIG. 4, the dish data includes data items of “dish”, “color feature amount”, and “shape feature amount”. In the data item “cooking”, a plurality of data can be set for the type of cooking described in FIG. For example, when the type of dish is A, even if the dish is the same as described above, the feature amount may differ greatly depending on the cooking method or the like. In the dish data, by recording a plurality of feature amounts for one type of dish, it is possible to prepare in advance dish data having a feature amount approximate to the image area of the dish included in the photographed image. The extraction accuracy of the cooking image area can be improved. The classification of “corresponding dishes” described in FIG. 3 may also be applied to the data item “dishes”.

  The “color feature value” is a data item that records a feature value related to the color of the dish, and the “shape feature value” is a data item that records a feature value related to the shape of the dish. Which priority value or combination of “color feature value” and “shape feature value” is used in determining a dish may be set according to the learning result in machine learning, for example. . For example, even in a dish that should be classified as the same dish (A1) in the present embodiment, there may be a difference in the feature amount due to variations in the cooking method. These differences can be recognized as the same dish by machine learning the characteristics of a plurality of captured images cooked in the same recipe, and the recognition accuracy of the dish can be improved.

[Operation of information processing device]
FIG. 5 is a flowchart illustrating an example of the operation of the information processing apparatus 10 according to the embodiment. FIG. 6 is a flowchart illustrating an example of tableware determination processing of the information processing apparatus 10 according to the embodiment.

  In FIG. 5, the information processing apparatus 10 determines whether an image has been acquired (step S11). Whether or not an image has been acquired can be determined, for example, by whether or not the image acquisition unit 101 has acquired a posted image of a dish transmitted from the client device 3. When it is determined that an image has not been acquired (step S11: NO), the information processing apparatus 10 repeats the process of step S11 and waits for acquisition of an image.

  On the other hand, when it is determined that an image has been acquired (step S11: YES), the information processing apparatus 10 acquires instrument data (step S12). Acquisition of appliance data can be executed, for example, when the detection unit 102 acquires appliance data recorded in the storage unit 105 or the like. In addition, when the determination information (for example, the product number etc. of tableware) which determines the tableware used for the posted recipe is included, you may make it acquire instrument data according to determination information.

  After performing the process of step S12, the information processing apparatus 10 performs a tableware determination process (step S13). Details of the tableware determination process will be described with reference to FIG.

  In FIG. 6, the information processing apparatus 10 acquires the shape feature quantity of tableware (step S131). The acquisition of the shape feature amount of the tableware can be executed, for example, when the detection unit 102 acquires data from the “shape feature amount” data item recorded in the storage unit 105 or the like.

  After executing the process of step S131, the information processing apparatus 10 acquires the pattern feature amount of the tableware (step S132). The acquisition of the pattern feature amount of the tableware can be executed, for example, when the detection unit 102 acquires data from the “pattern feature amount” data item recorded in the storage unit 105.

  After executing the process of step S132, the information processing apparatus 10 acquires the color feature amount of the tableware (step S133). The acquisition of the color feature value of the tableware can be executed, for example, when the detection unit 102 acquires data from the “color feature value” data item recorded in the storage unit 105 or the like.

  After executing the process of step S133, the information processing apparatus 10 acquires the texture feature amount of the tableware (step S134). The texture feature amount of the tableware can be acquired, for example, when the detection unit 102 acquires data from the “texture feature amount” data item recorded in the storage unit 105 or the like.

  After executing the process of step S134, the information processing apparatus 10 performs pattern matching of tableware (step S135). For example, the pattern matching of tableware is performed by the detection unit 102 using the image acquired in the process of step S11 for at least one data item from the acquired shape feature amount, pattern feature amount, color feature amount, or texture feature amount. This matching can be executed by extracting the image area of the tableware included in the image and determining the tableware. The pattern matching of the shape or the pattern is executed, for example, by enhancing the contour of the subject photographed by differentiating the acquired image and determining whether the contour shape approximates the shape feature amount or the pattern feature amount. be able to. By performing pattern matching of shapes, it is possible to determine an image area such as tableware included in a captured image. By performing pattern pattern matching on the image area of tableware extracted in shape pattern matching, the target area for pattern pattern matching can be reduced, so that the processing load can be reduced.

  In color pattern matching, determination can be made based on whether or not the color information of pixels included in the acquired image approximates the color feature amount within a predetermined range. Note that the processing load can be reduced by executing the color pattern matching on the image area of the tableware extracted in the shape pattern matching.

  Texture pattern matching can be determined based on whether or not the reflected light contained in the acquired image and the shadow caused by fine irregularities on the surface approximate the texture feature amount. Note that the processing load can be reduced by executing the texture pattern matching on the image area of the tableware extracted in the shape pattern matching.

  After executing the process of step S135, the information processing apparatus 10 ends the tableware determination process of step S13 illustrated in the flowchart. Note that the tableware determination process shown in FIG. 6 is performed when the pattern matching in step S135 is executed after the shape feature value, the pattern feature value, the color feature value, and the texture feature value of the tableware are acquired in steps S131 to S134. Illustrated. However, the tableware determination process may be executed by pattern matching based on each feature amount, for example.

  Moreover, the tableware determination process shown in FIG. 6 illustrated the case where it determines with the same algorithm, without distinguishing a container and cutlery. However, in the tableware determination process, for example, the determination of the container and the determination of the cutlery may be determined by different algorithms. For example, an algorithm for extracting a container from a captured image and an algorithm for extracting cutlery may be executed in parallel.

  In addition, since the shape of the container photographed as described above varies greatly depending on the photographed angle, the image tilt due to the photographing angle may be corrected. For example, a bowl such as a bowl is photographed in an oval shape when photographed from an oblique direction, but the image may be corrected by assuming that this is a circle.

  Returning to the description of FIG. After executing the process of step S13, the information processing apparatus 10 determines whether the tableware data has been successfully determined in the process of step S13 (step S14). The failure to determine tableware data means that the tableware (container) and the cutlery cannot be determined. If it is determined that the determination of tableware data has not been successful (step S14: NO), the information processing apparatus 10 ends the operation shown in the flowchart. That is, FIG. 5 shows the case where the dish image area extraction described later is not executed when the tableware data cannot be determined. Thereby, erroneous detection of the image area of the dish can be prevented and detection accuracy can be improved.

  On the other hand, if it is determined that tableware data has been successfully determined (step S14: YES), the information processing apparatus 10 determines whether tableware has been determined (step S15). When it is determined that the tableware has been determined (step S15: YES), the information processing apparatus 10 acquires correspondence data (dish arrangement data and tableware correspondence data) about the tableware (step S16). The acquisition of the correspondence data can be executed, for example, when the detection unit 102 acquires the correspondence data from the storage unit 105 or the like.

  On the other hand, when it is determined that tableware could not be determined (step S15: NO), or after executing the process of step S16, the information processing apparatus 10 determines whether or not cutlery has been determined (step S17). . When it is determined that the cutlery has been determined (step S17: YES), the information processing apparatus 10 acquires correspondence data (cutlery correspondence data) about the cutlery (step S18). The acquisition of the correspondence data can be executed, for example, when the detection unit 102 acquires the correspondence data from the storage unit 105 or the like.

  On the other hand, when it is determined that the cutlery cannot be determined (step S17: NO), or after executing the process of step S18, the information processing apparatus 10 determines the type of dish (step S19). The determination of the type of dish in step S19 is determined based on the corresponding dish for the determined tableware and the corresponding dish for the determined cutlery. For example, when the corresponding dishes for the dishes are “A, B, C” and the corresponding dishes for the cutlery are “A, B, C, D, E”, the detection unit 102 uses the common “A, B”. , C "is determined as the type of dish. Here, “D, E” which is not common may be determined by lowering the priority as a dish from which the image area is extracted.

  After executing the process of step S19, the information processing apparatus 10 acquires the feature amount of the dish determined in step S19 (step S20). The acquisition of the feature amount of the dish can be executed, for example, when the image extraction unit 103 acquires the dish data from the storage unit 105 or the like. As described with reference to FIG. 4, the feature amount of the dish is data in which a feature amount indicating the feature of the dish such as color and shape is set for the dish.

  After executing the process of step S20, the information processing apparatus 10 extracts an image area of the dish (step S21). For example, the image extraction unit 103 performs pattern matching with the feature amount of the dish data for the region on the captured image where the dish is expected to be arranged based on the dish arrangement data. It can be executed by extracting a continuous range in which the feature amount is approximated as an image region of the dish. Since the dish image area is adjacent to the tableware image area, the boundary may be determined in pattern matching based on the feature amount of the container in the corresponding arrangement.

  One or a plurality of areas may be extracted from the dish image area extracted in step S21. For example, when dishes are arranged so that they are scattered on a plate, or when main dishes and side dishes are arranged separately, dishes are extracted as multiple image areas on the plate. become. Note that the dish image area may exist outside the range specified by the dish arrangement data (corresponding arrangement) of tableware. For example, this is the case where the dishes are arranged so as to jump out of the container. If the image extraction unit 103 determines that the image area of the dish is not closed in the area specified by the corresponding arrangement, the image extraction unit 103 performs pattern matching in the captured image until the image area of the dish is closed. You may make it extract.

  After executing the process of step S21, the information processing apparatus 10 analyzes the cooking area (step S22). The analysis of the cooking area is a process of analyzing information obtained from the extracted cooking image area. For example, the cooking region analysis is performed by the cooking analysis unit 104 analyzing the size of the extracted region and analyzing how many people are cooking. In addition, the analysis of the cooking region may be performed by dividing the main dish and the garnish from the size, position, color, and the like of the extracted region when a plurality of cooking image regions are extracted.

  After executing the processing of step S22, the information processing apparatus 10 outputs the processing result (step S23) and ends the operation shown in the flowchart. The processing results to be output are, for example, the dish image area extracted in the process of step S21, the analysis result analyzed in the process of step S22, the type of dish determined in the process of step S19, and the like. The output of the processing result is, for example, output as display data to be displayed on the display device 21, data output to the server 4, or print output to a printer (not shown).

  5 and 6 show the case where the image area of the dish arranged on the tableware is extracted for one tableware. For example, a plurality of tableware are arranged on the table, With respect to the photographed images in which the number of items is photographed, it is possible to extract the image regions of the dishes arranged in the plurality of containers by executing the exemplified processing for each tableware.

[Specific examples of cooking image area extraction]
FIG. 7 is a diagram illustrating a first example of extraction of a cooking image area performed by the information processing apparatus according to the embodiment.

  In FIG. 7, FIG. 7A shows a square dish 1001 and its corresponding arrangement 1002. In the square dish device data, the shape, pattern, color, and texture feature quantities of the square dish are stored, and the square dish included in the photographed image can be determined. Corresponding arrangement 1002 indicates a position where a dish is likely to be arranged on the square dish 1001. A square dish is highly likely to have a long and narrow dish in Japanese cuisine, and the corresponding arrangement 1002 is set beyond the range of the dish in the longitudinal direction.

  FIG. 7B is a photographed image in which a salt-fried salted mackerel 1003 is arranged on a square dish 1001. As shown in the figure, a dish such as Sanma's grilled salt may be arranged so as to jump out of the square plate in the longitudinal direction. In this embodiment, instead of simply setting the corresponding arrangement in the shape of the container, it is possible to set the corresponding arrangement assuming a dish in which the container is used, so that the detection accuracy of the image range of the dish is improved. It becomes possible.

  FIG. 8 is a diagram illustrating a second example of cooking image area extraction executed by the information processing apparatus according to the embodiment.

  In FIG. 8, FIG. 8 (A) shows a Japanese food round dish 1011 and its corresponding arrangement 1012. The appliance data of the Japanese-style round plate 1011 stores the shape, pattern, color, and texture feature quantity of the round plate, respectively, and the Japanese round plate included in the photographed image can be determined. Corresponding arrangement 1012 indicates a position where a dish is likely to be arranged on Japanese dish 1011. The Japanese food round plate 1011 has a high possibility that dishes are arranged in the center of the plate in Japanese food, and the corresponding arrangement 1012 is set in a circle inside the outer diameter of the plate.

  FIG. 8B is a photographed image in which meat potatoes 1013 are arranged on a Japanese dish 1011. As shown in the drawing, the meat potatoes 1013 are evenly arranged on the plate without being biased to one place in the round plate. In the extracted image range of the dish, for example, by analyzing the area of the meat potato 1013 with respect to the size of the Japanese dish 1011, it is possible to analyze the number of dishes.

  FIG. 8C is a photographed image in which the omelet rice 1015 is arranged on the western dish 1014. Here, the feature amount of the Western dish 1014 is similar to that of the Japanese dish 1011 in shape and pattern (solid color). However, the Japanese round plate 1011 and the western round plate 1014 are greatly different in color and texture features. For example, the Japanese food round plate 1011 has a color unique to Japanese tableware and a rough texture. On the other hand, the western dish 1014 has a white, glossy and smooth texture. In the present embodiment, tableware is determined based not only on the shape but also on the features of the pattern, color, and texture, so the Japanese round plate 1011 and the western round plate 1014 are used as different types of dishes (dishes arranged). It is possible to define it with instrument data. Since western dishes such as omelet rice 1015 are arranged on the western dish 1014, it is easy to determine that the dish is omelet rice by setting the Western meal in advance for the corresponding dish.

  As described above, the information processing system according to the present embodiment acquires appliance data indicating the appearance of appliances that serve dishes, storage unit that stores tableware correspondence data indicating correspondence between appliances and dishes, and acquires images. An image acquisition unit, a detection unit that detects an appliance that is the same or similar to the appearance indicated by the appliance data from the acquired image, and a dish that is associated with the tableware correspondence data of the detected appliance is determined, and the determined dish An image extraction unit that extracts an image region from the image. With this configuration, the cooking portion included in the image can be accurately detected.

  The information processing method according to the present embodiment includes a storage step of storing appliance data indicating the appearance of an appliance that distributes dishes, tableware correspondence data indicating correspondence between the appliances and dishes, and an image acquisition step of acquiring an image. A detection step of detecting an appliance that is the same as or similar to the appearance indicated by the appliance data from the acquired image, a dish associated with the tableware correspondence data of the detected appliance is determined, and an image area of the determined dish is imaged And an image extracting step for extracting from the image. With this configuration, the cooking portion included in the image can be accurately detected.

  In addition, each step in the information processing method described in the present embodiment, or each process in the information processing program and the recording medium storing it, does not limit the execution order. For example, the execution order of the above steps may be executed in an arbitrary order.

  The information processing apparatus is recorded by recording a program for realizing the functions of the information processing apparatus 10 on a computer-readable recording medium, causing the computer system to read and execute the program recorded on the recording medium. 10 may be performed. Here, “loading and executing a program recorded on a recording medium into a computer system” includes installing the program in the computer system. The “computer system” here includes an OS and hardware such as peripheral devices. Further, the “computer system” may include a plurality of computer devices connected via a network including a communication line such as the Internet, WAN, LAN, and dedicated line.

  The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. As described above, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM. The recording medium also includes a recording medium provided inside or outside that is accessible from the distribution server in order to distribute the program. The code of the program stored in the recording medium of the distribution server may be different from the code of the program that can be executed by the terminal device. That is, the format stored in the distribution server is not limited as long as it can be downloaded from the distribution server and installed in a form that can be executed by the terminal device. Note that the program may be divided into a plurality of parts, downloaded at different timings, and combined in the terminal device, or the distribution server that distributes each of the divided programs may be different. Furthermore, the “computer-readable recording medium” holds a program for a certain period of time, such as a volatile memory (RAM) inside a computer system that becomes a server or a client when the program is transmitted via a network. Including things. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

  Further, a part or all of the information processing apparatus 10 in the above-described embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional unit of the information processing apparatus 10 may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the advancement of semiconductor technology, an integrated circuit based on the technology may be used.

  As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to In addition, one aspect of the present invention can be modified in various ways within the scope of the claims, and the technical aspects of the present invention also relate to embodiments obtained by appropriately combining technical means disclosed in different embodiments. Included in the range. Moreover, it is the element described in said each embodiment and modification, and the structure which substituted the element which has the same effect is also contained.

  DESCRIPTION OF SYMBOLS 1 ... Information processing system, 10 ... Information processing apparatus, 11 ... CPU, 12 ... RAM, 13 ... ROM, 14 ... HDD, 15 ... Display control part, 16 ... Input control part, 17 ... Communication I / F, 101 ... Image Acquisition unit, 102 ... detection unit, 103 ... image extraction unit, 104 ... dish analysis unit, 105 ... storage unit

Claims (8)

A storage unit that stores appliance data indicating the appearance of an appliance that distributes dishes, and tableware correspondence data that indicates correspondence between the appliances and the dishes;
An image acquisition unit for acquiring images;
A detection unit for detecting an instrument that is the same as or similar to the appearance indicated by the instrument data from the acquired image;
An information processing system comprising: an image extraction unit that determines the dish associated with the tableware correspondence data of the detected appliance and extracts an image area of the determined dish from the image. The storage unit further stores dish arrangement data indicating the correspondence between the appearance of the utensil when the dish is arranged on the utensil and the arrangement of the dish,
The information processing system according to claim 1, wherein the image extraction unit extracts the image area based on the dish arrangement associated with the dish arrangement data of the detected appliance. The storage unit further stores cutlery data indicating the appearance of cutlery, and cutlery correspondence data indicating correspondence between the cutlery and the dish,
The detection unit further detects a cutlery that is the same as or similar to the appearance indicated by the cutlery data from the acquired image,
The information processing system according to claim 1, wherein the image extraction unit further determines the dish associated with the cutlery correspondence data of the detected cutlery and extracts the image region. The storage unit stores texture data indicating the texture of the surface of the instrument as the instrument data,
The information processing system according to any one of claims 1 to 3, wherein the detection unit detects the appliance based on the stored texture data. The storage unit further stores feature amount data indicating a feature amount of a captured image of the dish,
The information processing system according to any one of claims 1 to 4, wherein the image extraction unit extracts the image region based on the stored feature amount. An acquisition unit that acquires appliance data indicating the appearance of an appliance that distributes dishes, tableware correspondence data indicating correspondence between the appliances and the dishes, and an image;
A detection unit for detecting an instrument that is the same as or similar to the appearance indicated by the instrument data from the acquired image;
An information processing system comprising: an image extraction unit that determines the dish associated with the tableware correspondence data of the detected appliance and extracts an image area of the determined dish from the image. A storage step of storing appliance data indicating the appearance of the appliance to which the dish is arranged, and tableware correspondence data indicating correspondence between the appliance and the dish;
An image acquisition step of acquiring an image;
A detection step of detecting an instrument that is the same as or similar to the appearance indicated by the instrument data from the acquired image;
An information extraction method comprising: an image extraction step of determining the dish associated with the tableware correspondence data of the detected appliance and extracting an image area of the determined dish from the image. A storage process for storing appliance data indicating the appearance of an appliance that distributes dishes, and tableware correspondence data indicating correspondence between the appliances and the dishes;
Image acquisition processing for acquiring images;
A detection process for detecting an instrument that is the same as or similar to the appearance indicated by the instrument data from the acquired image;
An information processing program for causing the computer to execute an image extraction process for determining the dish associated with the tableware correspondence data of the detected appliance and extracting an image area of the determined dish from the image.

Images