JP7064853B2 - Cooking content discrimination system and intake content discrimination system - Google Patents

Description

The present invention relates to a cooking content discriminating system for discriminating cooking content at home and an intake content discriminating system for discriminating intake content in a meal.

At a restaurant, a system has been proposed in which the contents of a meal ingested by a customer or a meal left by a customer are automatically determined by photographing a meal situation from a table (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2005-044170

When trying to apply the above-mentioned system to households, the ingredients may change from household to household, and it may not be possible to grasp the ingredients from the finished dish in a dish such as curry rice where the ingredients blend in. Therefore, even if the system proposed above is applied to a home, the accuracy is too low and it cannot be put into practical use as a system suitable for each home.

The present invention has been made in view of the above circumstances, and is an accuracy of a cooking content determination system capable of accurately and automatically determining the cooking content cooked at home, and an accuracy of the intake content ingested in a home meal. It is an object of the present invention to provide an intake content discrimination system that can be discriminated well and automatically.

In order to achieve the above object, the cooking content determination system of the first invention comprises a first photographing means for photographing a cooking situation of a dish in a kitchen, a second photographing means for photographing a cooking situation of a dish in a dining room, and the first photographing means. It is characterized by comprising (1) a cooking content discriminating means for discriminating cooking contents based on the photographing means and each photographing result by the second photographing means.

According to the first invention, it is possible to comprehensively determine the cooking content based on the cooking situation in the kitchen and the dining room. Therefore, even if a part of cooking is done in the kitchen and the rest of the cooking is done in the dining room, the cooking contents can be grasped accurately and automatically.

According to the second aspect of the present invention, in the first invention, the cooking content determining means is a means for determining the type and amount of ingredients and seasonings used for cooking from the results of photography by the first photographing means, and the first aspect. (2) A means for determining the type and amount of ingredients and seasonings used for cooking from the results of photography by the photographing means is provided.

According to the second invention, as the cooking content, in addition to the ingredients in the kitchen and the dining room, the seasoning can be grasped. Therefore, for example, even if condiments and spices are added to a dish once completed in the kitchen, the ingredients and seasonings used in the dish served in the meal are accurately and automatically grasped. be able to.

According to a third aspect of the invention, in the second aspect, the cooking content determining means includes a dish registered in a cooking database containing at least recipe information, calorie information, and nutrients related to various dishes, and the first photographing means and the above-mentioned first photographing means. Compared with the food photographed by the second photographing means, the difference portion of the ingredients and seasonings used is corrected, and the calories and nutrients of the photographed food are discriminated based on the corrected result. do.

According to the third invention, recipe information, calorie information, nutrients, etc. are based on a cooking database, and by using different ingredients used for each household as correction factors, the database can be utilized while utilizing the advantages of the cooking database. Can make up for the shortcomings of.

In the fourth aspect of the invention, in the first to third inventions, the cooking content discriminating means specifies a reference dimension based on an image taken by the first photographing means or the second photographing means, and the said. At least the amount of the foodstuff used for the cooking is determined based on the dimensions.

If the size of the same food cannot be determined, calories and the like cannot be calculated, but according to the fourth invention, the amount of food taken by each shooting means is specified from the standard dimensions reflected in the shot image. be able to.

A fifth aspect of the invention comprises a storage means for storing food data taken by the first photographing means and the second photographing means for at least a certain period of time in the first to fourth inventions, and the cooking content determination. If it is determined that the dish photographed by the first photographing means and the second photographing means is not the dish prepared by the present cooking, the means is the previously cooked dish stored in the storage means. If it is a previously cooked dish, the cooking content is determined based on the cooking status of the dish.

At home, there are cases where food is prepared and stored, or the remaining food is stored in a refrigerator or the like and used again for meals. According to the fifth invention, when a dish taken in the past is served for a meal in the dining room, it is possible to judge this situation and identify the ingredients, seasonings, etc. that have been identified in the past. There is. As a result, it is possible to deal with pre-made items that are often seen at home.

The intake content discrimination system of the sixth invention is a dish targeted for discrimination by the cooking content discrimination system from the cooking content discrimination system according to any one of the first to fifth inventions and the shooting result by the second photographing means. An intake amount discriminating means for discriminating the intake amount of the food, and an estimation means for estimating at least one of the calorie intake and the nutritional balance of the person who ate the food based on the discrimination results of the cooking content discriminating means and the intake amount discriminating means. , Equipped with.

According to the sixth invention, at least one of the calorie intake and the nutritional balance of the person who ate is estimated from the cooking content determined by the cooking content determination system and the intake status of the food provided by the cooking. .. By using this estimation result, it is possible to obtain various situations due to diet at home, such as excessive calories and imbalance in nutrition.

Top view showing the outline of the kitchen and the dining room. The schematic diagram which shows the structure of the discrimination system. Schematic diagram for explaining the registration information of a cooking database. A flowchart showing the shooting process. A flowchart showing a cooking content discrimination process. The schematic diagram which shows the data of the identified foodstuffs and the like. An explanatory diagram for explaining the correction process. A flowchart showing an intake content discrimination process. A flowchart showing the process for pre-preparation.

Hereinafter, an embodiment embodying the present invention will be described with reference to the drawings. FIG. 1 is a plan view showing an outline of a kitchen 11 and a dining room 12, and FIG. 2 is a schematic view showing a configuration of a discrimination system.

As shown in FIG. 1, the building 10 is provided with a kitchen 11 and a dining room 12 as indoor spaces. In the present embodiment, the kitchen 11 and the dining room 12 are formed as a continuous integrated space.

Kitchen equipment 21 is installed in kitchen 11, and kitchen equipment 21 separates kitchen 11 and dining 12. The kitchen equipment 21 is provided with a horizontally long workbench 22 (kitchen table), and a stove 23, a sink 24 (sink), and a counter 25 are provided on the upper surface of the workbench 22. The stove 23 and the sink 24 are arranged on both sides of the workbench 22 in the longitudinal direction (width direction), and a counter 25 is arranged between them. In the dining room 12, a dining set 28 having a dining table 26 and a plurality of chairs 27 is installed.

The kitchen 11 and the dining room 12 are equipped with a discrimination system having a cooking content discrimination function and an intake content discrimination function. The cooking content discrimination function captures the cooking process at home and discriminates the cooking content from the shooting result, and the intake content discrimination function captures the meal process in the dining room 12 and prepares a meal from the shooting result. It is intended to determine the intake content of the person who has taken the food. Hereinafter, the configuration of the discrimination system having these functions will be described in detail.

As shown in FIG. 2, a first camera 31 is installed on the ceiling surface 11a of the kitchen 11 as a first photographing means for photographing a cooking state of food in the kitchen 11. The first camera 31 is arranged above the kitchen equipment 21, and the shooting direction is directed downward so as to shoot the workbench 22. The first camera 31 is a camera capable of shooting a moving image, and the shooting range S1 is set so as to include the entire counter 25 and the stove 23 in order to shoot the cooking situation in the kitchen 11 without omission (FIG. 1).

On the ceiling surface 12a of the dining room 12, a second camera 32 is installed as a second photographing means for photographing the cooking state of the food in the dining room 12. The second camera 32 is installed based on the fact that a part of cooking is performed in the dining room 12, for example, the cooking is finished in the dining room 12, condiments and spices are added according to individual taste, and so on. It is a thing. The second camera 32 is arranged above the dining table 26, and the shooting direction is directed downward so as to shoot the top plate 26a of the dining table 26. The second camera 32 is a camera capable of shooting a moving image, and a shooting range S2 is installed so as to include the entire top plate 26a so that the second camera 32 can shoot anywhere on the top plate 26a when cooking is performed. See Figure 1).

The first camera 31 and the second camera 32 are electrically connected to the controller 35, and each camera 31 and 32 outputs an image signal according to the shooting content to the controller 35.

The controller 35 includes a well-known microcomputer including a CPU, ROM, RAM, and the like. The controller 35 is provided with a cooking database 36 in which information on ingredients, seasonings and dishes is registered in advance.

Here, the registration information of the cooking database 36 will be described with reference to FIG. The registered information of the cooking database 36 includes ingredient information 36a, seasoning information 36b, and cooking information 36c.

In the food material information 36a, image information of the food material, calorie information of the food material, and nutrient information indicating nutrients contained in the food material are registered in association with the type of the food material. Of these, the calorie information indicates the calories per standard amount of the foodstuff (for example, one potato in the case of potatoes).

Further, in the food material information 36a, reference size information indicating the size of each food material (for example, vegetables) whose amount is generally specified by the number of used food materials is registered. Further, for foodstuffs (for example, meat etc.) whose amount is generally specified by weight, conversion information for converting from size to weight is registered.

In the seasoning information 36b, the image information of the mark attached to the container of the seasoning, the calorie information, the nutrient information, and the conversion information are registered in association with the type of the seasoning. In the present embodiment, a unique mark (identification information) is attached to each seasoning container, and the image information of the above mark corresponds to this. This takes into consideration that there are seasonings that are difficult to distinguish on the image, such as sauce and soy sauce, salt and sugar. Further, the conversion information in the seasoning information 36b is information for converting the size in the case of a liquid substance and the number of grains in the case of a granular substance into the amount used in the unit of measurement (for example, 1 teaspoon). There is.

In the food information 36c, image information of food, recipe information, calorie information, and nutrient information are registered in association with the type of food. The recipe information indicates a general recipe of the dish, and the information indicating the amount of each ingredient used in the recipe information indicates an amount corresponding to, for example, four people. Further, the calorie information indicates the calories for one person of the dish prepared according to the recipe of the recipe information, and the nutrient information indicates the nutrients contained in the dish. Of these, at least recipe information can be displayed on a display terminal 40 (see FIG. 2) such as a smartphone or a PC, and it is possible to cook by looking at the recipe.

In the controller 35 shown in FIG. 2, the types and amounts of ingredients and seasonings used in cooking and cooking are performed based on the shooting results of the first camera 31 and the second camera 32 and the registration information of the cooking database 36. Determine the type of food, etc. Further, the intake amount of the food of the person who ate is determined from the image taken by the second camera 32, and the calorie intake and the nutrient balance are estimated based on the determination result of the foodstuff and the like and the determination result of the intake amount.

The estimated calorie intake and nutritional balance information is stored in the storage unit 38 provided in the controller 35. The storage unit 38 is composed of a non-volatile memory, and the calorie intake and the like stored in the storage unit 38 can be displayed on the display terminal 40. Further, the storage unit 38 stores the prepared food data (cooking history) for at least a certain period (for example, one week) from the cooking date. Further, the controller 35 is provided with a timer unit 39 having a timekeeping function, a clock function, and a calendar function.

Further, as shown in FIGS. 1 and 2, a first motion sensor 41 for detecting the presence of a person in the kitchen 11 is installed in the kitchen 11, and the presence of a person in the dining 12 is detected in the dining 12. A second motion sensor 42 for detecting is installed. Each of these motion sensors 41 and 42 is electrically connected to the controller 35 and outputs a detection signal to the controller 35.

Next, the shooting process executed by the controller 35 will be described with reference to the flowchart of FIG. This process is executed in a predetermined time cycle.

In the photographing process, first, in step S101, it is determined whether or not the presence of a person is detected in either the kitchen 11 or the dining 12 based on the detection signals of the motion sensors 41 and 42. If the presence of a person is not detected, the shooting process is terminated. On the other hand, when the presence of a person is detected, the image of the kitchen 11 is taken by the first camera 31 in step S102, and the image of the dining 12 is taken by the second camera 32 in step S103. .. The processes of steps S102 and S103 are executed in parallel, and the shooting processes of the cameras 31 and 32 are all executed continuously.

In the following step S104, each image taken by the first camera 31 and the second camera 32 is analyzed to extract the feature amount (color, contour, brightness), and the ingredients and seasonings (of the container) contained in the taken image are extracted. Mark) image is recognized. In addition, when cooking is nearing completion to some extent, the image of the dish included in the photographed image is recognized.

Next, in step S105, the detection signals of the motion sensors 41 and 42 are referred to, and it is determined whether or not the presence of a person is not detected in both the kitchen 11 and the dining room 12. That is, it is determined whether or not the person who was present in the kitchen 11 or the dining 12 has left the kitchen 11 or the like. When the presence of a person is detected, that is, when the person remains present, the process returns to step S102 and the processes of steps S102 to S104 are repeated.

On the other hand, when the presence of a person is not detected, that is, when the person leaves the kitchen 11 or the like, the shooting by the first camera 31 and the second camera 32 is stopped in step S106, and then the shooting process is performed. To finish.

Next, the cooking content determination process executed by the controller 35 will be described with reference to FIGS. 5 and 6. FIG. 5 is a flowchart showing the cooking content discriminating process, and FIG. 6 is a schematic diagram showing data of the discriminated ingredients and the like. This process is executed in a predetermined time cycle.

In the cooking content discrimination process, the type and amount of the ingredients and seasonings used in cooking are discriminated, the type of cooked dish, the amount to be prepared, and the amount to be provided are discriminated, and the calories and the like are estimated. Here, the case of making curry rice and salad as a dish will be described as an example.

As shown in FIG. 5, in the cooking content determination process, in step S201, it is determined whether or not the image of the foodstuff or the seasoning is recognized in step S104 in the photographing process of FIG. If the image of the foodstuff or the like is recognized, the process proceeds to step S202, and the photographed foodstuff or the like is collated with the image information registered in the foodstuff information 36a or the seasoning information 36b of the cooking database 36 to check the foodstuff or the like. Determine the type. The determination result of the foodstuff or the like is stored in the RAM of the controller 35.

In the following step S203, the amount of ingredients and seasonings used is determined based on the image recognition result of step S104. At this time, in the case of vegetables, etc., the number of pieces used (for example, 1/2 piece, 1 piece, etc.) is determined, in the case of meat, etc., the weight used (for example, 100 g, etc.) is determined, and in the case of seasonings, the amount used (for example, 100 g) is determined. For example, 1 teaspoon) is determined. The number of pieces used, the weight used, and the amount used are determined as follows, for example.

First, the number of pixels of the food material image in the captured image is grasped, and the number of pixels of the image of the reference object (for example, a cutting board placed on the counter 25 or one side of the dining table 26) whose dimensions are specified in advance is grasped. .. Next, the actual size (reference dimension) per pixel is specified from the number of pixels of the reference object image, and the actual size (reference dimension) is multiplied by the number of pixels of the food material image to derive the size of the food material. This is performed in each of the vertical and horizontal directions of the captured image to derive the vertical and horizontal dimensions of the foodstuff, and the size (area) of the foodstuff is derived from the result.

Then, when the food material is vegetables or the like, the derived size is compared with the reference size information (FIG. 3) registered in the food material information 36a of the cooking database 36, and the number of used pieces is derived. For example, when the number of ingredients recognized from the captured image is one, and the size of the ingredients is about ½ of the size indicated by the reference size information, the number of ingredients used is halved. It shall be an individual.

When the food material is meat or the like, the conversion information (FIG. 3) registered in the food material information 36a is used to derive the weight to be used from the above-derived size. The conversion information defines the weight (g) of the foodstuff per unit area (for example, 1 square centimeter), and when deriving the weight to be used, the size of the foodstuff is specified with respect to the weight specified in the conversion information. By multiplying, the weight per ingredient is obtained.

In the case of liquid seasoning, the conversion information (FIG. 3) of seasoning information 36b is used to derive the usage amount from the derived size, and in the case of granular seasoning, the above conversion information is used to derive the usage amount from the number of grains. Derive the usage. If the seasoning is measured and used with a measuring spoon or other measuring instrument, use it based on the size of the measuring instrument and the amount of seasoning charged into the measuring instrument (for example, about 1/2 of the measuring instrument). The quantity may be derived.

The usage amount of the foodstuff or the like determined as described above is stored in the RAM of the controller 35 in association with the type of the foodstuff or the like.

After the execution of step S203 or when a negative determination is made in step S201, it is determined in step S204 whether or not the food image is recognized in step S104 in the photographing process of FIG. When the image of the dish is recognized, in step S205, the photographed dish is collated with the image information registered in the dish information 36c of the dish database 36, and the type of the dish is determined.

In the following step S206, it is determined whether or not the dish determined in step S205 is a cooked dish. If it is determined that the dish is not cooked, the process proceeds to step S207 to determine the amount of dish to be prepared. In this step, the amount of each ingredient used determined from the photographed image is compared with the amount of each ingredient included in the recipe information of the cooking information 36c in the cooking database 36, and it is determined how many dishes are prepared. .. In addition, the amount of food prepared is not determined by comparing the amount of all the ingredients used in cooking, but the ingredients that are expected to have little variation in the amount used in each household (for example, curry roux in the case of curry rice). It is better to do it based on the amount comparison for the amount of water). In addition, the amount of food prepared may be determined from the size of the cooking utensil (pot, etc.) used for cooking.

In the following step S208, the amount of food served on the tableware, that is, the amount of food served (serving amount) is determined. This determination is made based on, for example, the size of the tableware and the ratio of the dish to the tableware. Specifically, first, by the same method as in step S203, the size of the tableware such as the plate on which the dish is served is determined, and the number of tableware for the number of people is specified. Then, on the image, the proportion of food in the tableware is obtained, and from the result, the amount of food is estimated for how many people. For example, if the proportion of food in a tableware of a size for one person is a certain amount (for example, about 70%), it is estimated as one person, and if it is about half of that, it is estimated as 1/2 person. This process is skipped if the dish is not served on the tableware.

After the execution of step S208 or when a negative determination is made in step S204, it is determined in step S209 whether or not cooking is completed. In this step, for example, from the shooting results of the cameras 31 and 32, it is determined whether or not the food has been left untouched for a predetermined period after being carried to the dining table 26. Then, when it is determined that the food has been left unattended, it is considered that cooking has been completed.

As a result of the above determination, if cooking is not completed, the process returns to step S201, and thereafter, the processes of steps S201 to S208 are repeated until cooking is completed. Of these, by repeating steps S201 to S203, as shown in FIG. 6, information on the ingredients used is accumulated in the RAM of the controller 35. That is, each time a new ingredient is used in cooking, the type of the ingredient is sequentially memorized together with the amount used.

Here, in the present embodiment, the type and amount of the ingredients and seasonings used in cooking are determined from both the image of the kitchen 11 and the image of the dining 12. Therefore, even when a part of cooking is performed in the kitchen 11 and the rest of the cooking is performed in the dining room 12, the ingredients and seasonings used for cooking can be grasped without omission. Therefore, for example, even when toppings such as boiled eggs and apples are added to the curry rice after being served on the tableware, the types and amounts of those additional ingredients can be appropriately grasped. (See FIG. 6).

On the other hand, when cooking is completed, the process proceeds to step S210, and a correction process for correcting the calories and the like of the prepared dish is executed according to the ingredients used and the amount used thereof. This treatment takes into consideration that the types and amounts of ingredients used differ from household to household, even for the same type of food. Here, the correction process will be described with reference to FIG. 7.

In the correction process, first, the prepared food is targeted, and the amount of each ingredient and each seasoning corresponding to one person is derived. Specifically, the amount of each ingredient used during cooking is divided by the amount of food prepared in step S207. For example, as shown in FIG. 6, when 400 g of pork diced meat is used for cooking and the cooking amount is determined to be for 4 people in step S207, the “cooking record” of FIG. As shown in the column, 100 g is derived as the amount of pork diced meat used.

Then, the amount of each ingredient and each seasoning derived is used as the amount of each ingredient and each seasoning specified in the recipe information of the cooking database 36 (the amount used for one person. The “recipe information” column in FIG. 7). To understand the difference. In the example of curry rice shown in FIG. 7, as the difference part, there is a difference regarding the amount of use such as "use 50 g more pork diced meat than the recipe" and "use rice, boiled egg and apple which are not in the recipe". The difference in the ingredients used in is grasped.

After that, the calories and nutrients of the ingredients and the seasonings in the difference portion are derived based on the ingredients information 36a and the seasonings information 36b of the cooking database 36. For example, in the case of pork diced meat, 50 g of calories (AAkcal) are derived by referring to the calorie information (FIG. 3) of the pork diced meat registered in the food material information 36a of the cooking database 36 (FIG. 7). See the "Corrected Calories" column).

Finally, the calories (calories for one person) and nutrients of the dish are read from the cooking information 36c of the cooking database 36, and the difference portion is corrected for the read calories and the like (the calories and nutrients of the difference portion are increased or decreased). do. This estimates the calories and nutrients of the food served.

The above correction process is performed for each dish whose type has been determined, that is, for each prepared dish. Therefore, when curry rice and salad are prepared, as shown in FIG. 7, correction processing is performed for each of them, and calories and nutrients are estimated respectively.

Returning to the description of the cooking content determination process of FIG. 5, after executing the above correction process, the cooking history is stored in the storage unit 38 in step S211. The cooking history contains information on the type of dish made, the calories and nutrients for one person in that dish. The cooking history is stored for a certain period from the cooking date, and then deleted. After the execution of step S211 the cooking content determination process is terminated.

In addition, toppings and seasonings may be added not only before meals but also during meals. When such additions (changes in the types and amounts of ingredients and the like) are made while eating, the types and amounts of the additional ingredients and the like are determined in steps S201 to S203 above, and then. In step S206 of the above, it is determined that the dish has been cooked. Then, if it is determined that the dish has been cooked, the process proceeds to step S210 to execute the correction process. As a result, calories and nutrients that reflect the ingredients added during the meal can be obtained.

Next, the intake content determination process executed by the controller 35 will be described with reference to the flowchart of FIG. This process is executed in a predetermined time cycle.

The intake content determination process determines the amount of food intake and estimates the calories and nutrients ingested by the person who ate the meal. In the intake content determination, first, in step S301, it is determined whether or not the food image is recognized in step S104 in the photographing process of FIG. If the image of the dish is recognized, the process proceeds to step S302, and the photographed dish is collated with the image information registered in the dish information 36c of the dish database 36 to determine the type of dish.

Next, in step S303, it is determined whether or not the determined dish is the dish created by this cooking. Specifically, the latest cooking history stored in the storage unit 38 is referred to, and it is determined whether or not the type of the dish indicated by the cooking history matches the type of the dish determined in step S302. ..

If the types of the determined dish and the dish in the cooking history do not match, that is, if the determined dish is not the dish produced by the current cooking, the process proceeds to step S304 to execute the pre-preparation process. This process is based on the case of eating prepared dishes and leftover dishes. Here, the pre-preparation process will be described with reference to the flowchart of FIG.

In the pre-preparation process, first, in step S401, the dish determined in step S302 of FIG. 8 is compared with the dish prepared within a certain period in the past. Specifically, all the cooking histories stored in the storage unit 38 are referred to, and it is determined whether or not there is a cooking history that matches the type of the determined dish. As already mentioned, the cooking history is deleted from the storage unit 38 after a certain period of time has passed from the cooking date, so this step determines whether or not the photographed dish was prepared within a certain period of time in the past. It will be judged.

As a result of the above determination, if there is a cooking history that matches the type of dish taken (YES in step S402), in step S403, information on calories and nutrients is obtained from the cooking history (past cooking data). Obtained and used as the calories and nutrients of the photographed dish. That is, it is assumed that the same kind of ingredients and seasonings as the dishes in the cooking history are used in the photographed dishes, and the cooking contents of the photographed dishes are determined. On the other hand, if there is no cooking history that matches the type of the photographed dish (NO in step S402), in step S404, the calories and nutrients of the same type of dish as the photographed dish are stored in the cooking database 36. It is obtained from the cooking information 36c (FIG. 3) of.

After the execution of step S403 or step S404, in step S405, the serving amount of the photographed dish is determined, and then the pre-preparation process is terminated. The process of step S405 is performed in the same manner as the determination of the serving amount of food in step S208 of FIG.

Returning to the explanation of the intake content discrimination process of FIG. 8, when the affirmative judgment is made after the execution of step S304 or in step S303 (when the photographed dish is the dish produced by this cooking), the dish is cooked in step S305. Determine the intake. The intake of food is determined based on the amount of food provided and the remaining amount of food. Specifically, first, the size of the tableware is determined from the shooting result, and the proportion of the dish in the tableware is determined on the image. Then, the remaining amount of the dish is determined from the size of the tableware and the ratio of the dish, and this is subtracted from the amount of the dish provided to derive the intake amount. For example, when the remaining amount of food is 1/4 serving and the serving amount is 1/2 serving, the intake amount for 1/4 serving is derived. As the serving amount of the dish, if the photographed dish is the dish prepared by this cooking, the serving amount determined in step S208 of FIG. 5 is used, and if it is not the dish cooked this time, the serving amount of FIG. 9 is shown. The provided amount determined in step S405 is used.

In the following step S306, the calorie intake of the person who ate is estimated based on the calorie of the dish and the intake of the dish. The calorie intake is the calorie intake determined in step S305 with respect to the calorie derived in step S210 of FIG. 5 (calorie of the dish for one person) when the photographed dish is the dish prepared this time. If it is not the dish cooked this time, it is derived by multiplying the calories acquired in step S403 or step S404 of FIG. 9 (calories of the dish for one person) by the above intake amount.

In the following step S307, the nutritional balance of the ingested food is estimated. If the photographed dish is a dish prepared by this cooking, the nutrients derived in step S210 of FIG. 5 are referred to, and if the dish is not the dish cooked this time, it was acquired in step S403 or step S404 of FIG. With reference to nutrients, the presence or absence of intake and the intake ratio of each nutrient are derived, and the nutritional balance such as vitamin deficiency is estimated.

After the execution of step S307 or when a negative determination is made in step S301 (when the image of the dish is not recognized), it is determined in step S308 whether or not the meal is completed. In this step, the movement of a person in the dining room 12 is recognized from the shooting result of the second camera 32, and it is determined whether or not the meal is finished based on the result.

If the meal is not completed, the process returns to step S301, and thereafter, the processes of steps S301 to S307 are repeated until the meal is completed. On the other hand, when the meal is completed, in step S309, the calorie intake estimated in step S306 and the nutritional balance estimated in step S307 are stored in the storage unit together with the type and date and time of the ingested dish. Remember in 38. As a result, information such as calorie intake is accumulated in the storage unit 38. Information such as calorie intake stored (stored) in the storage unit 38 can be viewed using the display terminal 40 (see FIG. 2) as described above. After the execution of step S309, the intake content determination process is terminated.

According to the present embodiment described in detail above, the following excellent effects are obtained.

・ The cooking situation of the dish was photographed, and the cooking content was determined from the photographed result. As a result, even if ingredients that are difficult to grasp from the finished dish are used, the ingredients used can be appropriately grasped. Moreover, a first camera 31 for the kitchen 11 and a second camera 32 for the dining 12 are provided, and the cooking situation of the dish is photographed by the respective cameras 31 and 32. As a result, even when a part of cooking is performed in the kitchen 11 and the rest of the cooking is performed in the dining room 12, the cooking contents can be accurately and automatically grasped.

-The structure is such that not only the ingredients but also the type and amount of seasonings are determined from the shooting results. Therefore, as the cooking content, in addition to the ingredients in the kitchen 11 and the dining 12, the seasoning can be grasped. As a result, for example, even when condiments and spices are added in the dining room 12 to the dish once completed in the kitchen 11, the types and amounts of the ingredients and seasonings used in the dishes served in the meal can be determined. It will be possible to grasp accurately and automatically.

-By comparing the dishes registered in the cooking database 36 with the dishes taken by the cameras 31 and 32, the differences in the ingredients and seasonings used can be grasped and registered in the cooking database 36. The calories and nutrients that have been taken are corrected according to the difference, and the calories and nutrients of the photographed dish are discriminated.

By registering the general calories and nutrients of each dish in the cooking database 36 in advance, it is possible to easily derive the calories and the like of the dish as long as the type of the photographed dish can be identified. On the other hand, the ingredients used that differ from household to household are not reflected, and there is a risk of inferior accuracy. In this respect, in this configuration, the calories and the like are corrected according to the difference between the registered dish and the photographed dish, so that the calories and the like reflecting the cooking contents of each household can be obtained. That is, it is possible to make up for the shortcomings of the database 36 while utilizing the advantages of the cooking database 36.

-Based on the images taken by the cameras 31 and 32, the reference dimensions are specified, and the dimensions of the photographed foodstuff are specified based on the dimensions. As a result, the size of the foodstuff used for cooking can be specified from the photographed image, and the amount of the foodstuff can be specified. In addition to the dimensions of the dining table 26 and the cutting board, the standard dimensions can be the dimensions of a ruler, a seal, printing, etc. that are installed in advance, and are not particularly limited.

-The data of the created dish is stored in the storage unit 38 for at least a certain period of time, and if the photographed dish is not the dish made by this cooking, the previously cooked dish stored in the storage unit 38 is used. It is determined whether or not there is a dish, and if it is a previously cooked dish, the cooking content of the photographed dish is determined based on the data of the dish.

At home, there are cases where food is prepared and stored, or the remaining food is stored in a refrigerator or the like and used again for meals. When eating these pre-prepared or leftover dishes, the cooking process does not occur immediately before the meal, so there is no opportunity to discriminate the ingredients, and there is a risk that calorie calculation and the like cannot be performed properly. In this respect, in this configuration, since the cooking content is determined from the data of the previously cooked food, it is possible to appropriately calculate the calories and the like even when eating the prepared food or the leftover food.

In this case, in the present embodiment, the above-mentioned fixed period is set to one week, and when a dish of the same type as the photographed dish is prepared within the past one week, the cooking content of the dish is photographed. It is configured to be used as cooking content. By limiting the reference period to a relatively short week in this way, if the types of dishes taken and those previously cooked match, those dishes will be prepared in the same cooking process. There is a higher possibility. Therefore, it is possible to accurately determine the cooking contents of the prepared and leftover dishes.

-The composition was such that the intake amount of food was discriminated from the photographed image of the dining 12, and the calorie intake and nutritional balance of the person who ate the meal were estimated based on the discrimination results of the intake amount and the cooking content.

According to this configuration, the calorie intake and nutritional balance of the person who ate the meal are estimated from the determined cooking content and the intake status of the food provided by the cooking. By using this estimation result, it is possible to obtain various situations depending on the meal at home, such as the fact that one meal is over-calorie and the nutritional balance is out of order. In addition, the above calorie intake and nutritional balance can be accumulated, and this can be used to diagnose the calorie acquisition status and nutritional balance status, and further, notification of the diagnosis result or notification based on the diagnosis result (notification of the diagnosis result or notification based on the diagnosis result). For example, you can give advice such as "Because there are many high-calorie meals, let's reduce calorie intake" and "Because we tend to be deficient in vitamins, let's eat a lot of vegetables and fruits"). This makes it possible to provide useful information for daily health management. By the way, it is good to notify regularly.

In the above embodiment, both the calorie intake and the nutritional balance are estimated, but only one of these may be estimated.

<Other embodiments>
(1) In the above embodiment, the information on the calorie intake and the nutrient balance is provided after the meal, but the information on the calorie and nutritional balance of the prepared dish or the provided dish is provided before or during the meal. It may be configured to be used. Specifically, the determination of the food intake is omitted, and the calorie and nutritional balance of the prepared food or the provided food are acquired from the correction result of step S210 in FIG. Then, based on the acquired calories and nutritional balance, advice (notification) such as "high calories" or "let's eat a lot of vegetables" is used by other notification means installed in the display terminal 40 or the dining room 12. )I do.

(2) In the above embodiment, the type of seasoning is determined based on the mark attached to the container of the seasoning, but the type of seasoning is determined based on the recipe information registered in the cooking database 36. It may be configured to be used. For example, when it is determined that a white seasoning is used in cooking, the amount used is compared with the recipe to determine that it is salt, or the order in which the white seasoning is used in cooking (timing). ) Can be compared with the recipe to distinguish it from salt. Further, when the white seasoning is only salt in the recipe of the dish, the seasoning may be discriminated as salt.

(3) In the above embodiment, each resident of the building 10 may be registered in advance in the cooking database 36 or the storage unit 38 of the controller 35, and the calorie intake and the nutritional balance may be accumulated for each resident. In this case, the determination of each resident can be performed by face authentication, ID authentication, or the like based on the captured images of the cameras 31 and 32. In addition, each resident is registered in association with the seat in the dining 12 (plural chairs 27), and which resident is from the position of the seat where the eater sits, which is taken by the cameras 31 and 32. It may be configured to determine whether or not.

(4) In the above embodiment, a plurality of cameras are provided in each of the kitchen 11 and the dining 12, and the vertical, horizontal, and depth dimensions of the food are specified from the shooting results of each camera, and the volume of the food is derived. It may be configured to determine the amount.

(5) In the above embodiment, the image information of the foodstuff or the like is registered in the cooking database 36, but instead of this, the information indicating the feature amount of each foodstuff or the like may be registered.

(6) In the above embodiment, the cooking database 36 may be provided in the web server on the Internet. That is, a communication unit that can be connected to the Internet is provided in the controller 35, and registration information of ingredients, seasonings, and dishes is obtained from the above-mentioned web server through the Internet. Further, the estimated calorie intake and nutritional balance may be stored in the above-mentioned web server or another web server and viewed from a mobile communication terminal such as a smartphone.

11 ... Kitchen, 12 ... Dining, 31 ... First camera as the first shooting means, 32 ... Second camera as the second shooting means, 35 ... Controller as the cooking content determination means, 36 ... Cooking database, 38 ... Memory Memory as a means.

Claims (4)

The first shooting method to shoot the cooking situation of food in the kitchen,
A second means of taking pictures of the cooking situation of food in the dining room,
A cooking content discriminating means for discriminating cooking content based on the results of each photographing by the first photographing means and the second photographing means, and
Equipped with
The cooking content determination means is
A means for determining the type and amount of ingredients and seasonings used for cooking from the results of photography by the first photography means, and
A means for determining the type and amount of ingredients and seasonings used for cooking from the results of photography by the second photography means, and
The dishes registered in the cooking database, which include at least recipe information, calorie information, and nutrients related to various dishes, are compared with the dishes taken by the first photographing means and the second photographing means, and are used. Means to correct the difference between ingredients and seasonings,
A means for determining the calories and nutrients of the photographed dish based on the corrected result, and
A cooking content discrimination system characterized by being equipped with . The cooking content determination means is
The first aspect of the present invention, wherein the reference dimension is specified based on the image taken by the first photographing means or the second photographing means, and at least the amount of the food material used for the cooking is determined based on the dimension. Cooking content discrimination system. The first shooting method to shoot the cooking situation of food in the kitchen,
A second means of taking pictures of the cooking situation of food in the dining room,
A cooking content discriminating means for discriminating cooking content based on the results of each photographing by the first photographing means and the second photographing means, and
A storage means for storing food data taken by the first photographing means and the second photographing means for at least a certain period of time, and a storage means .
Equipped with
The cooking content determination means is
If it is determined that the dish photographed by the first photographing means and the second photographing means is not the dish prepared by the present cooking, is it a previously cooked dish stored in the storage means? Judge whether or not,
In the case of a previously cooked dish, the cooking content discrimination system is characterized in that the cooking content is discriminated based on the cooking status of the dish. The cooking content determination system according to any one of claims 1 to 3 and the cooking content determination system.
An intake amount discriminating means for discriminating the intake amount of a dish targeted for discrimination by the cooking content discriminating system from the photographing result by the second photographing means, and an intake amount discriminating means.
An estimation means for estimating at least one of the calorie intake and the nutritional balance of a person who ate a meal based on the discrimination results of the cooking content discriminating means and the intake amount discriminating means.
Ingestion content discrimination system equipped with.

Images