JP7478698B2 - Apparatus, system, program and method for monitoring food intake and temperature-based feedback - Google Patents

Description

The present invention relates to a technology for collecting and analyzing information related to eating behavior, in particular information related to food and drink intake.

In recent years, with the increase in the elderly population and growing health consciousness, there have been many efforts to reassess daily eating habits and improve dietary life.

For example, Patent Document 1 discloses a dietary life management device that allows a user to check their own diet. This dietary life management device is equipped with a dietary life information transmission unit having a chopstick detection unit that detects when chopsticks are placed on the chopstick rest, another dietary life information transmission unit that is installed in the chew count measuring device and has a chewing information detection unit, and a dietary life information processing unit that has control means for collecting and processing dietary life information from these dietary life information transmission units.

Patent Document 2 also discloses a dietary management program that allows a user to check their own diet. This dietary management program causes an information and communication terminal, which is a computer equipped with a camera unit, to execute dietary management processing that includes the steps of: acquiring image data with a camera unit; recognizing images of eating utensils from the acquired image data by image recognition processing; detecting dietary information, which is information related to the user's diet, from the recognized images of the eating utensils; and collecting and processing the detected dietary information.

Furthermore, Patent Document 3 discloses a chopstick movement characteristic determination device that can determine how chopsticks are moving. Specifically, this device is equipped with a chopstick movement characteristic determination unit that calculates relative acceleration from the fixed chopstick acceleration and the working chopstick acceleration, and determines which of multiple predetermined movement characteristics the fixed chopstick and the working chopstick are moving with when the relative acceleration is calculated, based on a relative acceleration pattern that shows the difference in relative acceleration for each axial direction. This makes it possible to measure eating behavior from the acceleration of the chopsticks, and also makes it possible to correct the way chopsticks are held.

Patent Document 4 also discloses tableware for promoting mastication that is equipped with a timer, making it possible to maintain a certain chewing time when eating. Specifically, when chopsticks are used as tableware for promoting mastication, a timer is started when the chopsticks reach a certain inclination when bringing food to the mouth, and while the timer is running, a display on the chopsticks displays a message to continue chewing, encouraging the user to maintain the chewing time.

Furthermore, Patent Document 5 discloses a food and drink intake management method that uses image recognition technology to detect when a user is about to bring food or drink to their mouth from a database of prohibited foods and drinks based on the user's health condition, mainly due to food and drink interactions with medications, and issues a warning when the user is about to eat or drink prohibited foods or drinks.

Patent Literature 6 also discloses a cooking assistance method that reduces the risk of food poisoning from each meal provided to each meal recipient when multiple meals are cooked simultaneously and individually by machine. This cooking assistance method specifically includes the steps of selecting one serving tray, measuring the target dish temperature, which is the surface temperature of each dish on the one serving tray, and measuring the central temperature of the ingredients in each dish, confirming that the central temperature of the ingredients in each dish satisfies the target food temperature condition, which is the temperature condition to be satisfied during cooking, measuring the surface temperature of each dish on the other serving tray, and determining that the ingredients in the dish whose surface temperature has been measured satisfies the target food temperature condition if the surface temperature of each dish on the other serving tray satisfies a predetermined condition based on the target dish temperature for the dish placed in the same position on the one serving tray.

Furthermore, Patent Document 7 discloses an information processing device that allows customers at a restaurant to eat food at an optimal temperature. Specifically, this device includes a storage means that stores sensor identification information that identifies a temperature sensor capable of detecting the temperature of food and drink provided to customers at the restaurant in association with customer identification information that identifies the table or seat of the customer to whom the food and drink was provided, a communication means that is capable of communicating with a store terminal provided in the restaurant or a table terminal provided at the table or seat and is capable of receiving the output value of the temperature sensor together with the sensor identification information, and a control means that transmits to the store terminal, based on a change in the received output value, a notification to encourage eating and drinking of the food and drink corresponding to the temperature sensor indicated by the sensor identification information received together with the output value, before the temperature of the food and drink changes to a temperature that is unsuitable for eating.

Furthermore, Patent Document 8 discloses a food or beverage serving method that enables restaurants to serve the next dish or beverage in accordance with the progress of each customer's meal while reducing labor costs. Specifically, this method includes an imaging step for imaging the food or beverage on the table in the restaurant, and an image information providing step for providing the store with image information captured by the imaging means as information to be used for determining the timing for the store to inquire about the availability of additional services from the customer.

As one embodiment of this food or beverage serving method, a method has been proposed in which an infrared camera is used to measure the temperature of the plate on the table to assist in timing the serving of food at the right temperature, and when the temperature of the plate becomes the same as room temperature, it is determined that the customer has no intention of eating and the waiter or other person is advised to serve the next dish at the right time.

JP 2014-097223 A JP 2015-146168 A JP 2015-225277 A JP 2000-270993 A JP 2018-181385 A JP 2020-010835 A JP 2018-128818 A JP 2004-252497 A

As explained above, various technologies have been proposed to monitor eating behavior and, based on the information obtained, realize optimal meals and promote the health of diners. However, these conventional technologies still cannot adequately grasp the actual contents of the food and drink provided, especially the temperature, and provide accurate feedback to diners.

For example, the technologies disclosed in Patent Documents 1 and 2 can obtain information about what food and drink is consumed, how long it takes, and how many times it is chewed, but cannot obtain information about the temperature of the food and drink. The same is true for the technologies disclosed in Patent Documents 3 and 4, which can collect various information from the movement of chopsticks while eating, such as the timing of feedback, but do not anticipate obtaining information about the temperature of food and drink.

Furthermore, the technology disclosed in Patent Document 5 also uses image recognition technology to identify the type of food and drink and can provide information mainly about food and drug compatibility, but it is still unable to obtain information about the temperature of the food and drink.

On the other hand, the technology disclosed in Patent Document 6 does indeed make it possible to obtain temperature information of food and drink using a thermal imaging camera and generate information to reduce the risk of food poisoning. However, it is not possible to recognize the contents of the provided food and drink other than its temperature, such as its type, and it is not envisaged to carry out temperature control appropriate to the contents of the food and drink, such as its type. In fact, with this technology, processing is carried out using given target food temperature conditions.

The technology disclosed in Patent Document 7 is similar in this respect, and although it is possible to detect the temperature of food and drink provided to customers in a restaurant, it does not recognize details of the food and drink provided other than its temperature, such as its type. Therefore, for example, when sending a food and drink promotion notification that takes into account the menu, information from a pre-set menu information database is required.

Furthermore, the technology disclosed in Patent Document 8 also makes it possible to detect the temperature of food and drink served to customers in a restaurant, but does not recognize anything other than the temperature of the food and drink served, such as its type. As a result, even when automatically cooking the next dish or drink, preparing to serve it, or serving it, it is necessary to use information from a food database or an order content recorder for each dining table.

In actual meals, the various foods and drinks provided usually have an appropriate temperature range for each type. For example, consuming warm foods and drinks whose temperatures are within the appropriate temperature range not only provides a delicious and comfortable meal, but also contributes to improving the health of the diner. In other words, it is expected to have a great effect of warming a cold body and improving metabolism and immunity. However, in the conventional technologies described in Patent Documents 6 to 8, it is impossible to provide feedback that takes into account the contents of the provided food and drink, for example, the appropriate temperature range for each type, unless knowledge of the contents is secured in advance.

The present invention aims to provide an eating behavior monitoring device, system, program, and method that can grasp the temperature of the food or drink provided according to its contents and provide more accurate feedback to the diner.

According to the present invention, a food and drink temperature acquisition means acquires food and drink temperature information, which is information related to the temperature of the food and drink, from a temperature measurement means capable of measuring the temperature of the food and drink involved in a meal of a diner;
a food and drink content acquisition means for acquiring food and drink content information, which is information related to at least the type of the food and drink for the diner , from a recognition means capable of recognizing at least the type of the food and drink and the amount of the food and drink;
eating behavior information generating means for receiving eating behavior related information from a storage means storing eating behavior related information that is information related to the past eating behavior of the diner, the information including at least information related to the temperature of the food or drink of the type when the food or drink was ingested, and generating eating behavior information by compiling the eating behavior related information;
An eating behavior monitoring device is provided that has a feedback means for predicting information relating to the future temperature of a type of food and drink based on the food temperature information and at least the food and drink content information relating to the type using a preset feedback standard or model based on the temperature and content of the food and drink, and for selecting or generating and providing feedback information including information relating to the timing, time and/or order in which the type of food and drink should be ingested, adjusted to suit the eater, based on the predicted information relating to the future temperature and information relating to the temperature of the type of food and drink when ingested, which is included in the eating behavior information.
According to the present invention, there is also provided a food and drink temperature acquisition means for acquiring food and drink temperature information, which is information relating to the temperature of the food and drink, from a temperature measurement means capable of measuring the temperature of the food and drink involved in a meal of a diner;
a food and drink content acquisition means for acquiring food and drink content information, which is information related to at least the type of the food and drink for the diner, from a recognition means capable of recognizing at least the type of the food and drink and the amount of the food and drink;
a feedback means for predicting information on a future temperature of the food and drink of the type based on the food and drink temperature information and at least the food and drink content information related to the type, using a preset standard or model of feedback based on the temperature and content of the food and drink, selecting or generating feedback information including information on the timing, time and/or order in which the food and drink of the type should be consumed based on the predicted information on the future temperature, generating control information for controlling an operation of cooling and/or heating the food and drink by a cooling device and/or a heating device previously mounted on the tableware related to the meal of the diner, based on the information on the timing, time and/or order in which the food and drink of the type should be consumed, and transmitting the control information to the cooling device and/or the heating device;
An eating behavior monitor is provided having the following:

In one embodiment of the present invention , the food temperature acquisition means also acquires environmental temperature information, which is information related to the temperature of the dining environment of the person eating, from the temperature measurement means or from another temperature measurement means,
It is also preferable that the feedback means predicts information relating to the future temperature of the food or drink of the type based on the environmental temperature information as well.

In one embodiment of the present invention, the eating behavior information includes information related to the time when the food or drink of the type was ingested.
It is also preferable that the feedback means adjusts information relating to the timing, time and/or order in which the type of food or beverage should be consumed, contained in the feedback information, to suit the diner, based on information relating to the time when the type of food or beverage was consumed.

Furthermore, as another embodiment of the present invention for generating eating behavior information , the eating behavior information also includes information related to the chewing behavior when the eater ingests the food of the type, which is generated from an output of a pre-installed electromyographic sensor capable of detecting the chewing of the eater,
It is also preferable that the feedback means adjusts information regarding the timing, time and/or order in which the food or beverage of that type should be ingested, contained in the feedback information, to suit the diner, based on information regarding the chewing behavior when the food or beverage of that type is ingested.

Furthermore, as yet another embodiment of the present invention for generating eating behavior information , the eating behavior information includes an eating behavior quality index for the diner, which includes an index relating to the appropriate temperature of the provided food and drink, an index relating to the degree to which the diner finished ingesting the provided food and drink while it was at an appropriate temperature, an index relating to the degree to which the diner finished eating while the food and drink was at an appropriate temperature, an index relating to the number of types of food and drink provided, and/or an index relating to the appropriate temperature of the diner's eating environment,
It is also preferable that the feedback means adjusts information relating to the timing, time and/or order in which the type of food or drink should be consumed, contained in the feedback information, to suit the diner, also based on the eating behavior quality index.

In addition, in the eating behavior monitoring device according to the present invention, it is also preferable that the temperature measuring means is a thermograph installed in the eating environment of the diner, a thermometer mounted on the utensils used by the diner during the meal, and/or a thermometer mounted on the tableware used by the diner during the meal.

Furthermore, as yet another embodiment of the present invention , it is preferable that the feedback means generates control information for controlling the irradiation of light onto the type of food or beverage by a pre-installed light irradiation means capable of irradiating the food or beverage, based on information regarding the timing, time and/or order in which the selected or generated type of food or beverage should be consumed, and outputs or transmits the control information to the light irradiation means.

Furthermore, as yet another embodiment of the present invention , it is also preferable that the feedback means generates control information for controlling a notification operation to the diner by a light source pre-mounted on the utensils used by the diner for the meal, a vibrator pre-mounted on the utensils, a light source pre-mounted on the tableware related to the diner's meal, and/or a vibrator pre-mounted on the tableware, based on information regarding the timing, time and/or order in which the selected or generated food or drink should be consumed, and transmits the control information to the light source and/or vibrator.

Furthermore, in the eating behavior monitor according to the present invention, it is also preferable that the recognition means acquires information related to an image including the food or drink from a pre-installed camera capable of photographing the food or drink, and recognizes at least the type and amount of the food or drink based on the information related to the image. Here, it is also preferable that the eating behavior monitor is an information processing device equipped with a recognition means capable of receiving information related to the image from an externally installed camera, a tablet computer equipped with such a camera and recognition means, a smartphone equipped with such a camera and recognition means, or an augmented reality (AR) type smart glass equipped with such a camera and recognition means.

According to the present invention, there is also provided a temperature measuring means capable of measuring the temperature of food and drink consumed by a diner;
a food and drink temperature acquisition means for acquiring food and drink temperature information, which is information relating to the temperature of the food and drink, from a temperature measurement means;
A recognition means capable of recognizing at least the type of the food and drink among the type and amount of the food and drink;
a food and drink content acquisition means for acquiring food and drink content information, which is information related to at least the type of the food and drink of the diner , from the recognition means;
eating behavior information generating means for receiving eating behavior related information from a storage means storing eating behavior related information that is information related to the past eating behavior of the diner, the information including at least information related to the temperature of the food or drink of the type when the food or drink was ingested, and generating eating behavior information by compiling the eating behavior related information;
An eating behavior monitoring system is provided that has a feedback means for predicting information relating to the future temperature of a type of food and beverage based on the food and beverage temperature information and at least the food and beverage content information relating to the type using a preset feedback standard or model based on the temperature and content of the food and beverage, and for selecting or generating and providing feedback information including information relating to the timing, time and/or order in which the type of food and beverage should be ingested, adjusted to suit the eater, based on the predicted information relating to the future temperature and information relating to the temperature of the type of food and beverage when ingested, which is included in the eating behavior information.

According to the present invention, there is further provided a program for causing a computer to function for monitoring an eating behavior of a diner, comprising:
a food and drink temperature acquisition means for acquiring food and drink temperature information, which is information relating to the temperature of the food and drink, from a temperature measurement means capable of measuring the temperature of the food and drink involved in a meal of a diner;
a food and drink content acquisition means for acquiring food and drink content information, which is information related to at least the type of the food and drink for the diner , from a recognition means capable of recognizing at least the type of the food and drink and the amount of the food and drink;
eating behavior information generating means for receiving eating behavior related information from a storage means storing eating behavior related information that is information related to the past eating behavior of the diner, the information including at least information related to the temperature of the food or drink of the type when the food or drink was ingested, and generating eating behavior information by compiling the eating behavior related information;
An eating behavior monitor program is provided that causes a computer to function as a feedback means that predicts information regarding the future temperature of a type of food and drink based on the food and drink temperature information and at least the food and drink content information relating to the type using a preset standard or model for feedback based on the temperature and content of the food and drink, and selects or generates and provides feedback information including information regarding the timing, time and/or order in which the type of food and drink should be ingested, adjusted to suit the diner, based on the predicted information regarding the future temperature and information regarding the temperature of the type of food and drink when ingested, which is included in the eating behavior information.

According to the present invention, there is further provided an eating behavior monitoring method in a computer for monitoring an eating behavior of a person eating a meal, comprising the steps of:
a step of acquiring food and drink temperature information, which is information relating to the temperature of the food and drink, from a temperature measuring means capable of measuring the temperature of the food and drink involved in the diner's meal, acquiring food and drink content information, which is information relating to at least the type of the food and drink, of the diner , from a recognition means capable of recognizing at least the type and amount of the food and drink, receiving eating behavior related information, which is information relating to the diner's past eating behavior, from a storage means that stores eating behavior related information including at least information relating to the temperature of the food and drink of that type when it was ingested, and generating eating behavior information that compiles the eating behavior related information ;
An eating behavior monitoring method is provided which includes a step of predicting information relating to the future temperature of a type of food and beverage based on the food temperature information and at least the food and beverage content information relating to the type using a preset standard or model for feedback based on the temperature and content of the food and beverage, and selecting or generating and providing feedback information including information relating to the timing, time and/or order in which the type of food and beverage should be ingested, which is adjusted to suit the diner, based on the predicted information relating to the future temperature and information relating to the temperature of the type of food and beverage when ingested, which is included in the eating behavior information.

The eating behavior monitoring device, system, program, and method of the present invention make it possible to grasp the temperature of the food or drink provided according to its contents, and provide more accurate feedback to the diner.

1 is a schematic diagram showing an embodiment of an eating behavior monitoring system including an eating behavior monitor according to the present invention; 11 is a graph for explaining an example of a temperature prediction processing result in the food and drink temperature prediction unit according to the present invention. 1 is a table for explaining one aspect of the contents of a food and drink optimum temperature database (DB) according to the present invention. 1 is a table for explaining one aspect of the contents of a food and drink combination DB according to the present invention. 1 is a flow chart illustrating an embodiment of a method for monitoring eating behavior including calculation of a QoF index according to the present invention.

The following describes in detail an embodiment of the present invention with reference to the drawings.

[Eating behavior monitoring system, device]
FIG. 1 is a schematic diagram showing an embodiment of an eating behavior monitoring system including an eating behavior monitor according to the present invention.

The eating behavior monitoring system of this embodiment shown in FIG.
(A) a visible light camera 2 capable of photographing food and drink placed on a dining table and generating image (video) information including the food and drink;
(a) a thermography camera 3 capable of photographing food and drink placed on a dining table and generating infrared image (video) information including the food and drink;
(c) a projector 4 capable of projecting feedback information (including a spotlight-like image (video) designating the food or drink) at a designated position among the positions of the food or drink placed on the dining table; and
(D) Chopsticks 5 with notification function, which are equipped with a thermometer at the end where food is picked up and at the opposite end, and further have a built-in vibrator and a light for notification;
(E) a myoelectric measuring device 6 that is attached to the head of the person eating the meal and that can measure the person's chewing movements;
(f) An eating behavior monitor device 1 capable of obtaining information from a visible light camera 2, a thermographic camera 3, chopsticks with notification function 5, and an electromyographic measuring device 6, for example, via short-range wireless communication such as Bluetooth (registered trademark) or Wi-Fi (registered trademark) or via wired communication, and providing "feedback information" regarding the eating behavior of the diner.

In this embodiment, the eating behavior monitor 1 has a food and drink recognition unit 112 that can analyze the image information acquired from the visible light camera 2 described above (A) using known image recognition technology and recognize one or both of the type and amount of each food and drink placed on the table (in this embodiment, at least the type).

In this embodiment, the eating behavior monitor 1 also has a food and environmental temperature acquisition unit 111 that can analyze the infrared image information acquired from the thermographic camera 3 (a) above using known thermographic analysis technology and generate information related to the temperature of each food and drink placed on the table and the surrounding environmental temperature. Note that in this embodiment, this food and environmental temperature acquisition unit 111 can also acquire information related to the temperature of each food and drink and the surrounding environmental temperature from the chopsticks with notification function 5 (d) above.

Furthermore, in this embodiment, the eating behavior monitor 1 can provide the above-mentioned (f) "feedback information" as a display on the display (DP) 107, as a sound from the speaker 109, by having the projector 4 project the feedback information onto a specific food or drink (via short-range wireless communication or wired communication), and even by activating a vibrator or a light on the chopsticks with notification function 5.

To provide this "feedback information," the eating behavior monitor 1 has the following features:
(A) the food and drink temperature acquisition unit 111 described above, which acquires "food and drink temperature information" that is information related to the temperature of the food and drink involved in the meal of a diner from a temperature measurement means (in this embodiment, a thermographic camera 3 or chopsticks with notification function 5) that can measure the temperature of the food and drink involved in the meal of the diner;
(B) a food and drink content acquisition unit 113 that acquires “food and drink content information” that is information related to one or both of the type and amount of the food and drink (at least the type in this embodiment) from the food and drink recognition unit 112;
(C) It has a feedback unit 115 that selects or generates and provides “feedback information” related to the eating behavior of the diner based on “food and drink temperature information” and “food and drink content information” using a preset feedback standard or model based on the temperature and contents of the food and drink.

In this way, the eating behavior monitor device 1 can obtain "food and drink temperature information" about the food and drink, not just by itself, but together with "food and drink content information" about the food and drink, making it possible to select or generate "feedback information" that takes both into consideration. In other words, it is possible to grasp the temperature of the provided food and drink according to its content, and provide more accurate feedback to the diner.

For example, the eating behavior monitor device 1 can grasp the type and temperature of each food and drink served, including the soup, and generate "feedback information" to the effect that for health or a suitable diet, the hot soup, whose temperature is within the optimum range, should be consumed first. It can then have the projector 4 project a spotlight-like image (video) onto the soup as feedback information, display the message "Please have the soup first," on the display 107, and further output the audio message "Please have the soup first" from the speaker 109.

Here, the "type" of food and drink in the "food and drink content information" in (B) above may be, for example, a dish name (menu name) such as miso soup or steak, or a genre such as noodles or soup. For example, a genre may be used for food and drink that is difficult to identify as a dish name (menu name) by the food and drink recognition unit 112. Furthermore, the "amount" of food and drink may be the amount of each food and drink, but instead of or in addition to that, it may also be the number of food and drink, for example the number of plates or dishes.

In addition, in this embodiment, the feeding behavior monitor device 1 is also capable of generating "feeding behavior information" including a QoF (Quality of Feeding behavior) index by using myoelectric signal information as feeding behavior-related information acquired from the myoelectric measurement device 6 described above in (e). Here, the QoF index, which will be described in detail later, is an index devised by the inventor of the present application and is an index for evaluating the quality of feeding behavior by taking into account the temperature of the food and drink ingested.

In this embodiment, the eating behavior monitor 1 is a tablet computer that can be easily placed on a dining table as needed, and for example, a camera that is standardly installed on this tablet may be used instead of or in combination with the visible light camera 2 to obtain image information.

As a modification, the food and drink recognition unit 112 may be provided in the visible light camera 2 or in an external information processing device to which the visible light camera 2 is connected, and the eating behavior monitor device 1 may receive and acquire the "food and drink content information" from there.

As a further modification, the eating behavior monitor device 1 can be an augmented reality (AR) type smart glass equipped with a camera (capable of photographing food and drink during a meal) and a food and drink recognition unit 112. In this case, "feedback information" can be displayed on a display placed in front of the eyes, mapped onto the food and drink on the dining table. It is also possible to give these AR type smart glasses the functionality of an electromyography device 6. This allows the diner to provide information that is the basis of "eating behavior information" simply by wearing these AR type smart glasses, which makes it possible to enjoy more accurate "feedback information."

Furthermore, although this eating behavior monitoring system has the above configurations (A) to (F) in this embodiment, it is not essential to have all of them, and it is sufficient if the system has at least a temperature measuring means capable of measuring the temperature of the food and drink, and a recognition means capable of recognizing the type and/or amount of the food and drink. Of course, if the recognition means requires image information including the food and drink, a camera capable of photographing the food and drink will also be required.

Here, we will discuss some general knowledge about the importance of the temperature of food and drink during meals. As the saying goes, food is medicine, and a healthy diet is essential for maintaining and improving health. The basis of a healthy diet is that hot foods should be eaten while hot, and cold foods while they are still cold.

It is said that, in terms of human taste, the lower the temperature of food or drink consumed, the less sweet and umami flavors are perceived, whereas bitter and salty flavors tend to be more readily perceived. Therefore, there is an optimum temperature range for finding food delicious, as a balance between these flavors, and the optimum temperature range is generally considered to be 60-70°C for hot food and 5-12°C for cold food.

Furthermore, there are many foods that must be consumed at the appropriate temperature to be delicious, such as hot soups and stews, piping hot ramen and pork buns, as well as cold tofu, cold juice and beer. As explained above, the temperature of food and drink consumed during meals is indeed an important parameter that must be monitored to ensure that a meal is both delicious and healthy.

[Device configuration, food behavior monitoring program]
Also in the functional block diagram of Figure 1, the eating behavior monitoring device 1 according to the present invention has a communication interface 101, a food and drink temperature information storage unit 102, a food and drink image information storage unit 103, an eating behavior related information storage unit 104, an optimum food and drink temperature database (DB) 105, a food and drink combination DB 106, a touch panel display (TP/DP) 107, a microphone 108, a speaker 109, and a processor memory.

Here, this processor memory stores one embodiment of the eating behavior monitor program according to the present invention, and also has computer functions, and performs eating behavior monitoring processing by executing this eating behavior monitor program. For this reason, the eating behavior monitor device according to the present invention may be a tablet computer as in this embodiment, but it can also be a wearable device such as a cloud server, a non-cloud server, a personal computer (PC), a smartphone, a notebook computer, or even an augmented reality (AR) type smart glass, equipped with the eating behavior monitor program according to the present invention, and of course it may also be a device dedicated to monitoring eating behavior.

The processor memory further includes a food and environmental temperature acquisition unit 111, a food and beverage recognition unit 112, a food and beverage content acquisition unit 113, an eating behavior information generation unit 114 including a QoF (Quality of Feeding Behavior) calculation unit 114a, a food and beverage temperature prediction unit 115a, a notification means control information generation unit 115b, and a food and beverage temperature control information generation unit 115c, a communication control unit 121, and an input/output control unit 122. These functional components can be considered as functions of an eating behavior monitor program stored in the processor memory. The process flow shown by connecting the functional components of the eating behavior monitor device 1 in the functional block diagram of FIG. 1 with arrows can also be understood as one embodiment of the eating behavior monitoring method according to the present invention.

Also in the functional block diagram of FIG. 1, the food and environment temperature acquisition unit 111 acquires food and drink temperature information from either or both of the thermographic camera 3 and the chopsticks with notification function 5, and also acquires environmental temperature information, which is information related to the temperature of the dining environment of the diner. In addition, food and drink temperature information and environmental temperature information may be acquired, for example, from a thermometer attached to the tableware related to the diner's meal or an indoor thermometer. In any case, the information related to the temperature of food and drink and the information related to the temperature of the dining environment collected from such temperature measurement means via the communication interface 101 and the communication control unit 121 is, for example, first stored in the food and drink temperature information storage unit 102, and the food and environment temperature acquisition unit 111 preferably receives this information as appropriate and adjusts and manages it as food and drink temperature information and environmental temperature information.

Here, the thermographic camera 3 is capable of outputting the temperature of each pixel in an infrared image within the angle of view every second, for example, as time-series temperature information. The food and environment temperature acquisition unit 111 receives the time-series temperature information from the food and drink temperature information storage unit 102, and calculates the temperature of each pixel by a known thermographic analysis technique.
(a) A group of pixels having a temperature higher (or lower) than the temperature of the surrounding pixels by a predetermined amount is regarded as a single piece of food or drink, and the highest (or lowest) temperature in the group of pixels regarded as a single piece of food or drink is regarded as the temperature of the food or drink. Furthermore, information linking this temperature with the identifier of the food or drink and the time is regarded as food or drink temperature information of the food or drink.
(b) The temperature of the pixels surrounding the group of pixels deemed to be food or drink (for example, the temperature of the surface of a dining table) can be regarded as the environmental temperature, and information linking this environmental temperature with time can be regarded as environmental temperature information.

In addition, the food and drink/environmental temperature acquisition unit 111 may, for example,
(c) The number of groups of pixels regarded as one food or drink is set as the number of types of food or drink (e.g., the number of dishes or plates),
(d) The number of pixels that make up one piece of food or drink can also be used as the amount of that food or drink (e.g. the amount served on a plate), and this information may be provided to the food or drink content acquisition unit 113 described later as food or drink content information.

In addition, the food and drink/environmental temperature acquisition unit 111
(e) information relating to the maximum or minimum temperature (the temperature that has changed the most from the ambient temperature) among the temperatures obtained from the thermometer at the tip of the chopsticks with notification function 5 that hold the food or drink is regarded as the temperature of the food or drink, and information linking this temperature with an identifier relating to the order (of the peak or valley) and time is regarded as food or drink temperature information of the food or drink (that is to be consumed using the chopsticks with notification function 5);
(f) It is also preferable to use the temperature (e.g., a temperature equivalent to the air temperature) obtained from a thermometer at the end of the notifying chopsticks 5 opposite to the end used to hold food as the environmental temperature, and to use information linking this environmental temperature with time as environmental temperature information. For example, a single piece of intake event information combining the food and drink temperature information and the environmental temperature information may be generated as (intake time, (intake order identifier,), food and drink identifier, food and drink temperature, environmental temperature). It is also possible to recognize that the diner's intake behavior of one bite has occurred at a single maximum or minimum peak or valley in the time-series information related to temperature obtained from the notifying chopsticks 5.

Furthermore, as a preferred embodiment of obtaining temperature information about food and drink using the notification function-equipped chopsticks 5, for example, each food and drink (recognized by the food and drink recognition unit 112 described below) is designated and displayed in sequence on the display 107 of the eating behavior monitor device 1, and the diner, upon seeing this, measures the temperature of each food and drink (at the time of ingestion) by touching the designated food and drink in sequence with the notification function-equipped chopsticks 5 before eating. In this case, a setting may be made to skip temperature measurement for food and drink that is clearly at room temperature.

The final food and drink temperature information (or environmental temperature information) may be determined, for example, by taking a weighted average between the food and drink temperature information (or environmental temperature information) obtained from the notification chopsticks 5 and the food and drink temperature information (or environmental temperature information) obtained from the thermographic camera 3. In this case, the correspondence between the foods and drinks for which the average is to be taken may be determined, for example, using the analysis results of the image (video) information from the visible light camera 2 (described later) (for example, the order of intake or time of intake, etc.).

Furthermore, it is also preferable that the thermometer mounted on the chopsticks 5 with notification function is a small temperature sensor with good temperature response. In order to improve the temperature response, the main body of the chopsticks 5 with notification function is made of a material with a relatively low thermal conductivity, such as titanium or stainless steel, among metals that generally have high thermal conductivity. The heat of the food or drink is easily transferred to the thermometer, but the thermal transfer between the thermometer (food or drink) and the gripping part of the chopsticks 5 may be limited. Furthermore, it is also possible to adopt a spoon or fork instead of or in combination with the chopsticks 5 as a eating tool equipped with a thermometer. For example, a known small strain sensor may be built into a spoon equipped with a thermometer to measure the weight of the food or drink to be consumed. This measured weight of the food or drink is included in the food and drink content information of the food or drink.

Also in the functional block diagram of FIG. 1, the food and drink recognition unit 112 acquires image (video) information including food and drink that is first stored in the food and drink image information storage unit 103 from the visible light camera 2 via the communication interface 101 and the communication control unit 121, and based on the image information, recognizes one or both of the type and amount of each food and drink placed on the table (in this embodiment, at least the type) using known image recognition technology (for example, using a trained food and drink type identification model). Of course, it is also possible to recognize the number of types of food and drink placed (the number of dishes or plates).

Here, the food and drink recognition unit 112 may acquire image (video) information including food and drink generated by a camera mounted on the eating behavior monitor device 1 placed on the dining table, and recognize one or both of the type (e.g., dish name) and amount (e.g., number of dishes/plates or amount served on each plate) of each food and drink (in this embodiment, at least the type).

Furthermore, it is also preferable that the food and drink recognition unit 112 compares the acquired image information with the infrared image information acquired from the thermographic camera 3, and based on the positions of corresponding pixels between the two, identifies in the image information the food and drink that corresponds to the food and drink (and its temperature) determined from the infrared image information, and assigns an identifier of the food and drink that is common to the food and drink temperature information to the recognized food and drink (type).

The food and drink content acquisition unit 113 acquires from the food and drink recognition unit 112 information on one or both (in this embodiment, at least the type) of type (e.g., dish name) and amount (e.g., number of dishes/plates or amount served on each plate) of each food and drink with an identifier as food and drink content information. The food and drink recognition unit 112 may determine not only the type of each food and drink but also the nutritional components of each food and drink (e.g., using a nutritional component database prepared in advance), and the food and drink content acquisition unit 113 may acquire food and drink content information including the nutritional component information of each food and drink.

Also in the functional block diagram of FIG. 1, the eating behavior related information storage unit 104 receives food and drink temperature information from the food and environment temperature acquisition unit 111, and food and drink content information from the food and drink content acquisition unit 113. It also receives myoelectric signal information (e.g., information linking the measurement time to a preprocessed myoelectric signal) from the electromyography device 6 via the communication interface 101 and the communication control unit 121, and stores this information as food and drink related information that is information related to the eater's past eating behavior (including at least food and drink temperature information).

The eating behavior information generating unit 114 receives the eating behavior related information from the eating behavior related information storage unit 104 and generates "eating behavior information" by summarizing the eating behavior related information. Specifically, this "eating behavior information" can be, for example, an eating behavior log in which the details of the eating behavior of each person for each meal are organized and recorded. The details of this eating behavior can include:
(a) the temperature of the food environment;
It is also preferable that (b) the time each food or drink was ingested, the order in which the foods and drinks were ingested, the meal duration (the start time and end time of the meal), as well as the total number of chews, the chewing pace, and the total number of bites, and (c) the type (name of dish), temperature, amount ingested, the time taken for one bite, and the number of chews for each food or drink ingested are recorded.

Here, the chewing-related information (b) and (c) above may be determined from the acquired myoelectric signal information by a biosignal processing method invented by the present inventor, for example, as described in JP 2020-192002 A, JP 2019-115410 A, JP 2019-107067 A, and JP 2018-139630 A, or by other known chewing measurement methods. In addition, from the eating behavior information, which is a long-term eating behavior log, it is possible to obtain information such as the temperature range at which a diner prefers to ingest food and drink, and the relationship between the temperature of the ingested food and drink and the environmental temperature.

In this embodiment, it is also preferable that the QoF calculation unit 114a of the eating behavior information generation unit 114 generates a "QoF (Quality of Feeding behavior) index" as information included in the eating behavior information from the food and drink information as described above. This QoF index is an index devised by the inventor of the present application, and is an index for evaluating the quality of eating behavior taking into account the exact temperature of the food and drink ingested. The calculation of this QoF index will be explained in detail later using FIG. 5.

<Feedback methods>
Also in the functional block diagram of FIG. 1, the feedback unit 115 uses a preset feedback “standard” or “model” based on the temperature and contents of the food or drink to:
(a) food and drink temperature information acquired from the food and drink/environmental temperature acquisition unit 111;
(b) Based on the food and drink content information acquired from the food and drink content acquisition unit 113, feedback information regarding the eating behavior of the diner is selected or generated and provided.

Here, the feedback information provided may be selected by searching from a feedback database that stores various types of feedback information in advance. In this case, this feedback database stores feedback information data organized by items related to the temperature and contents of food and drink, and can be considered as a feedback provision "model" that can output corresponding feedback information data using search keys related to the temperature and contents of food and drink.

The feedback information provided may be generated by reading out various feedback information templates, which are organized by items related to the temperature and contents of the food and drink, based on information related to the temperature and contents of the food and drink as a "criterion" for reading, and applying the relevant feedback information to the template. Furthermore, the feedback information provided may be determined using a trained machine learning "model" that outputs an identifier for the feedback information to be provided based on the temperature and contents of the food and drink.

In addition, in this embodiment, each food or drink that is assigned its own temperature in the food and drink temperature information and each food or drink that is assigned its own type and amount in the food and drink content information are assigned a common identifier as described above. Therefore, the feedback unit 115 can determine the type (e.g., dish name), temperature, and amount of each food or drink that has been served from the food and drink temperature information and the food and drink content information.

(Feedback using predicted temperature information)
As one suitable feedback process, the feedback unit 115 may predict information related to the future temperature for each food or drink (for each type of food or drink), and select or generate feedback information including at least one of information related to the timing, time, and order in which the food or drink (type) should be consumed based on predicted temperature information, which is the predicted information related to the future temperature. Here, when making this prediction, it is also preferable to predict information related to the future temperature of the food or drink based on the environmental temperature information included in the food and drink temperature information, and generate the predicted temperature information. Hereinafter, one aspect of the process of generating predicted temperature information performed by the food and drink temperature prediction unit 115a of the feedback unit 115 will be described.

In general, the temperature T(t) of food or drink that is initially at T0 and is exposed to an environmental temperature Te after t seconds is expressed by the following formula (1): T(t) = (T0 - Te) x exp(-t/τx) + Te
Here, τx is the thermal time constant in the set unit time. The above formula (1) can be transformed into a form for calculating the thermal time constant τx as follows:
(2) τx＝－t／ln{(T(t)－Te)／(T0－Te)}
Therefore, if the initial temperature T0 of the food or drink, the temperature T(t) after t seconds have elapsed, and the environmental temperature Te are known, the thermal time constant τx of the food or drink can be determined using the above formula (2). Furthermore, by using this determined thermal time constant τx and the above formula (1), the temperature of the food or drink after a predetermined time (e.g., 30 minutes) can be calculated to generate predicted temperature information.

The thermal time constant τx can be calculated without using formula (2) above, for example, if a database is prepared in which thermal time constant values for each type of food and drink are recorded in advance. However, the actual thermal time constant value is often affected by the physical properties of the ingredients and the physical properties of the tableware used, even for the same dish, and may vary from the previously recorded value. Therefore, it is preferable to determine the thermal time constant τx using the above method.

Figure 2 is a graph illustrating an example of the temperature prediction processing results in the food and drink temperature prediction unit 115a.

Figure 2 shows a graph of predicted temperature information for hot food A and cold food B placed in an ambient temperature (30°C), i.e., the relationship between predicted temperature (°C) and elapsed time (minutes). Both predicted temperatures show an exponential change approaching the ambient temperature (30°C). Here, the optimum temperature range for cold food B is set to 0-15°C, while the optimum temperature range for hot food A is set to 50-75°C.

Therefore, from the state of the temperature changes of each food, it can be seen that initially, cold food B is within the optimum temperature range, but hot food A exceeds the optimum temperature range and is too hot, and furthermore, cold food B exceeds the optimum temperature range before hot food A enters the optimum temperature range. Therefore, in this case, the feedback unit 115 can select or generate feedback information to the effect that, for example, "it is recommended to consume cold food B before hot food A."

Regarding the temperature of food and drink, for example, the food and drink is stirred each time it is ingested, which changes the heat distribution and thus the measured temperature of the food and drink. Also, the food and drink is often torn each time it is ingested, which changes the amount and shape of the food and drink, which changes the thermal time constant of heat release and absorption. Therefore, it is preferable that the food and drink temperature prediction unit 115a re-performs the temperature prediction process from the newly measured temperature of the food and drink after the intake behavior occurs, rather than relying forever on the initial temperature prediction result. For example, the prediction of the time during which the temperature of the food and drink will be within the optimum temperature range may be updated every predetermined time (at a predetermined cycle) based on the temperature of the food and drink at that time.

In addition, even if no ingestive behavior is detected, when the food and beverage temperature prediction unit 115a detects an "outlier" or "change point" in the change in the temperature of the food or beverage that exceeds a predetermined error of the temperature measurement means, it is preferable to perform the temperature prediction process again from the newly obtained temperature of the food and beverage.

By the way, as an "outlier", if the "difference" between the predicted value and the actual measured value of the temperature of food or drink deviates by more than a predetermined threshold when compared with the variation of the difference over a predetermined period of time in the past, the actual measured value of this "difference" can be considered an outlier. For example, the root mean square error (RMSE) can be used as the variation of the difference over a predetermined period of time in the past, and if the "difference" exceeds twice the value of the RMSE as a predetermined threshold, the actual measured value of this "difference" can be considered an outlier. Furthermore, it is also possible to determine whether or not it is an "outlier" from the relationship of distance to the past k actual measured values and the relationship of distance to the latest actual measured value using the k-nearest neighbor method. In any case, both "outliers" and "change points" can be detected by known statistical methods.

Returning to Figure 1, the specific details of the feedback processing in the feedback unit 115 will now be explained.

For example, when the feedback unit 115 determines that the served food or drink A is too hot beyond its appropriate temperature range, it may select or generate feedback information such as "Food or drink A should not be consumed until it cools down a little more (for example, until it is below 70°C), or should be consumed with caution." By providing such feedback information to the diner, it is possible to prevent burns in the oral cavity and even future oral cancer due to repeated heat stimulation to the oral cavity.

The feedback unit 115 may also select or generate feedback information that notifies the diner of the appropriate temperature range for each food or drink served. This allows the diner who receives this feedback information to consume each food or drink at an appropriate temperature, and thus enables the diner to practice delicious and healthy eating.

In this embodiment, the feedback unit 115 can use the type of food or drink (e.g., dish name, menu name, or genre) as a search key to obtain the optimum temperature range of the food or drink from the optimum food and drink temperature DB 105. As shown in FIG. 3, the optimum food and drink temperature DB 105 includes:
(a) a menu and ingredient optimum temperature DB in which, for each menu name, the genre of the dish, the ingredients, and the optimum temperature range for that menu name are recorded and managed in association with each other, and (b) a genre optimum temperature DB in which, for each cuisine genre, the optimum temperature range for the food and drink of that genre is recorded and managed in association with each other.
It is equipped with the following.

Incidentally, in the case of a menu name such as "meat bun" where the internal temperature is much higher than the external temperature and it is difficult to determine whether or not it is within the optimum temperature range by measuring the temperature from the outside, it is possible to register "difficult to predict" in the relevant column of the menu/ingredients optimum temperature DB, as shown by "difficult" in the "prediction" column of the menu/ingredients optimum temperature DB in Figure 3. Also, if both the dish name and genre information are reliably obtained in the processing of the food and drink recognition unit 112, it is more likely that the optimum temperature range information in the menu/ingredients optimum temperature DB is more accurate than the optimum temperature range information in the genre optimum temperature DB, so it is also preferable to set it so that the optimum temperature range information in the menu/ingredients optimum temperature DB is used as feedback information.

For example, when the feedback unit 115 determines that the difference between the temperature of the delivered food and drink B, which is originally warm, and the ambient temperature is less than a predetermined threshold (e.g., 10°C), it may select or generate feedback information such as "Food and drink B has gone cold, so you should stop consuming it," since continuing to consume cold food and drink can be stressful.

Furthermore, the feedback unit 115 may select or generate feedback information suggesting which food or drink should be consumed first based on the type of each food or drink served. For example, the feedback unit 115 may select or generate feedback information suggesting that "a vegetable salad (rich in dietary fiber to prevent overeating and a sudden rise in blood sugar level and prevent lifestyle-related diseases) should be consumed first" or feedback information suggesting that "a warm soup should be consumed first (to warm the body and increase diet-induced thermogenesis, a type of metabolism, and promote digestion, absorption, and metabolism)."

For example, the feedback unit 115 may use the ingredients of each food or drink served as a search key to obtain food combination information between the foods and drinks from the food and drink combination DB 106, and select or generate feedback information regarding food combinations based on this food combination information. Here, as shown in FIG. 4, the food and drink combination DB 106 includes an ingredient combination DB that records and manages, for each pair of ingredients 1 and 2, the efficacy, effects, and preventable diseases resulting from consuming both ingredients (of the foods and drinks) together. Also, the food ingredients used as the search key here can be obtained from the food and drink optimum temperature DB 105 described above, using the type of the food or drink as a search key.

For example, the feedback unit 115 may identify food and drink C and food and drink D that go well together and are both within the appropriate temperature range at that time among the foods and drinks served, and select or generate feedback information to the effect that "food and drink C and food and drink D should be consumed together (to increase nutrient absorption) and before food and drink E, which is still too hot."

Furthermore, the feedback unit 115 may select or generate feedback information related to a warning that "the consumption of food or drink F may pose health problems" for food or drink F that is too hot (or too cold) beyond a predetermined temperature range. Also, feedback information that encourages consumption behavior that is considered favorable from a health perspective may be selected or generated, taking into consideration both the temperature of the food or drink and the environmental temperature. For example, feedback information that "it is cold this morning, so you should consume hot soup first" or feedback information that "it is hot today, so you should consume cold tofu first" may be selected or generated.

For example, the feedback unit 115 may predict the change in temperature of the food and drink over time from the environmental temperature and the temperature of the food and drink in the food and drink temperature prediction unit 115a described above, and may select or generate feedback information such as "since the room temperature is currently below 20°C, hot food and drink G should be consumed within 15 minutes." Furthermore, for example, it is also possible to use as feedback information information on how many minutes it will take for ice cream that has become too cold and solidified to reach the appropriate temperature, or information on how many more minutes this ice cream will remain at the appropriate temperature.

(Feedback using eating behavior information)
Next, a mode of selecting and generating feedback information using the "eating behavior information" generated by the eating behavior information generating unit 114 will be described.

Specifically, it is also preferable that the feedback unit 115 adjusts information regarding the timing, time, and/or order in which a certain type of food or drink should be ingested, which is included in the eating behavior information, to suit the diner, based on information regarding the temperature of the food or drink when the diner ingests the food or drink, which is included in the eating behavior information. Furthermore, if this eating behavior information also includes intake time information regarding the time when the food or drink of the same type was ingested, the feedback unit 115 may adjust information regarding the timing, time, and/or order in which the food or drink of the same type should be ingested, which is included in the feedback information, to suit the diner, based on this intake time information as well.

Furthermore, if this eating behavior information also includes information related to the chewing behavior when ingesting that type of food or drink, which is generated from the output of the electromyography device 6 capable of detecting the chewing of the diner, it is also preferable that the feedback unit 115 adjusts the information related to the timing, time and/or order in which that type of food or drink should be ingested, which is included in the feedback information, to suit the diner, based on the information related to the chewing behavior when ingesting that type of food or drink.

Here, we will explain a specific example of feedback processing using the eating behavior information mentioned above.

For example, if the feedback unit 115 determines from the eating behavior information that a diner has a tendency to have a sensitive tongue and consume a type of food or drink whose optimum temperature range is a temperature range that includes 75°C only after it has cooled to below 65°C, the optimum temperature range for that type of food or drink for that diner may be changed to a temperature range below 65°C and registered in the optimum food and drink temperature DB 105. Also, if such a tendency to consume food or drink when it is colder is observed only in the summer, for example, the optimum temperature range for that type of food or drink in the summer may be changed to a temperature range below 65°C and registered in the optimum food and drink temperature DB 105.

For example, when the feedback unit 115 determines from the eating behavior information that the average mealtime of a diner is below a predetermined threshold, or when the feedback unit 115 determines that the average number of chews per mouthful of a diner is below a predetermined threshold, it may select or generate feedback information such as "eat slowly, chew well, and relax." Furthermore, when the feedback unit 115 determines from the eating behavior information that the standard deviation of mealtimes exceeds a predetermined threshold, it may select or generate feedback information such as "eat regularly."

In addition, when the feedback unit 115 determines from the eating behavior information that the average number of times per day that the diner consumes hot foods and beverages (foods and beverages with a predetermined optimum temperature range or higher) is below a predetermined threshold, it is preferable to select or generate feedback information such as "you should actively consume hot foods and beverages."

Above, we have explained how to select or generate feedback information using "eating behavior information." Next, we will explain the process of selecting or generating feedback information using the QoF index when the "eating behavior information" includes the above-mentioned QoF index.

Specifically, the feedback unit 115 may adjust information on the timing, time and/or order in which food and drink should be ingested, which is included in the feedback information, to suit the diner, based on the diner's QoF index, which is included in the "eating behavior information." For example, when feedback information on food and drink P is selected or generated for a diner whose "optimum temperature intake index (described below)" for food and drink P is below a predetermined threshold, it is possible to add information to the feedback information such as "You should try to wait until this food and drink P has cooled (or warmed) before ingesting it."

<Calculation of QoF index>
Here, the specific content of the QoF index will be described. As the QoF index, at least one of the following indexes, and preferably all of them, can be adopted.

(Optimal temperature index)
As an indicator of the appropriate temperature of the food and drink provided,
(3) (Proper Temperature Serving Index) = (Number of types of food and beverages served at the proper temperature) / (Total number of types of food and beverages (number of dishes))
Here, the optimum temperature provision index takes a value in the range of 0 to 1.

(Optimal Temperature Intake Index)
As an index of the degree to which the provided food and drink P was consumed while still at the appropriate temperature,
(4) (Optimal temperature intake index of food and drink P) = (intake time of food and drink P taken within the optimal temperature range) / (total intake time of food and drink P)
Here, the optimal temperature intake index also has a value in the range of 0 to 1.

(Dietary index at optimal temperature)
As an indicator of the degree to which people finish their meal while food and drink are still at the appropriate temperature,
(5) (optimum temperature dietary index) = Σ _i (optimum temperature intake index for food and drink i) / (total number of food and drink types (number of dishes))
Here, the optimal temperature diet index also takes a value in the range of 0 to 1. This optimal temperature diet index is the average value of the optimal temperature intake index for each food and drink served in one meal.

(Food and drink type number index)
As an indicator of the number of types of food and drink provided,
(6) (Food and beverage type number index) = (total number of food and beverage types (number of dishes)) / (predetermined target type total number (target number of dishes))
Here, if the total number of food and drink types (number of dishes) exceeds a predetermined target total number of food and drink types (target number of dishes), the food and drink type number index may be rounded to 1.

(Environmental Optimum Temperature Index)
As an index of the appropriate temperature of the user's eating environment,
(7) (Environmental Optimal Temperature Index)=(Time during which the environmental temperature was within a predetermined optimal temperature range)/(Total meal time)
Here, the optimum environmental temperature index also has a value in the range of 0 to 1. It is also preferable that the "predetermined optimum temperature range" of the environmental temperature is set to a value range according to the season.

It is also possible to adopt QoF indices other than those described above. For example,
(a) information on the number of people around the dinner table determined based on image information acquired from the visible light camera 2 using a known face image recognition technique;
(b) Using publicly known voice recognition technology, a communication index indicating whether or not communication occurred during the meal may be determined based on voice information acquired from a microphone and information regarding the amount of conversation that occurred during the meal.

moreover,
(a) information related to the number of occurrences of keywords that indicate a good meal (e.g., "Delicious!"), determined based on voice information acquired from a microphone using known voice recognition technology; and
(b) Information relating to the time and number of times a person “smiles” during a meal, determined based on myoelectric signal information acquired from the myoelectric measuring device 6 using the technology disclosed in JP 2020-048917 A, an invention by the present inventor; and
(c) It is also possible to determine a satisfaction index indicating whether or not the patient was satisfied with the meal from information regarding the amount of intake, which is determined based on the electromyographic signal information obtained from the electromyographic measuring device 6.

The QoF indexes described above may be displayed as radar charts for each of the above-mentioned indexes on, for example, the display 107 immediately after the meal is finished, and presented to the diner. It is also preferable to present the average values of the QoF indexes for a given period in the past (for example, one day, one week, or one month) as, for example, a bar graph or line graph, so that the diner can recognize the changes in the QoF indexes to date. Furthermore, for example, information related to such QoF indexes of each elderly diner may be shared with external medical institutions, etc., and in particular, when the diner is an elderly person living alone, it may be preferable to encourage the local community, local government, or regular medical institution to make suggestions such as recommending online dining.

Figure 5 is a flowchart outlining one embodiment of the eating behavior monitoring method according to the present invention, including calculation of the QoF index. Of course, the eating behavior monitoring method according to the present invention is not limited to the form shown in the flowchart.

(S101) The type (dish name) and number of types (number of items) of the served food and drink are obtained (or updated).
(S102) The suitable temperature range and ingredient names for each type (each dish name) are searched for using the suitable food and drink temperature DB 105.
(S103) Combinations between foods and drinks are searched for using the food and drink combination DB 106 based on the (searched) ingredient names for each type.
(S104) The temperature of each type (food name) and the environmental temperature are measured and managed.
(S105) Using the measurement results from step S104, the future temperatures of each type (food name) are predicted.

(S106) Using the information obtained in steps S102 to S105,
(a) feedback information regarding the timing, time, and order of intake of each food and drink;
(b) selecting or generating feedback information including at least one of information regarding food and drink combinations, and (c) information regarding precautions for intake;
(S107) Feedback information is provided.

(S108) The amount of food and drink of each type (food name) is measured and monitored.
(S109) Ingestion behavior (e.g., number of chews, etc.) is recorded and managed.
(S110) (If deemed necessary) feedback information (such as a warning) is provided regarding the detected intake behavior.
For example, when an eating action of holding a plate in one's hand is detected, if the temperature of the food or drink on the plate is above (below) the optimum temperature range, feedback information stating that "this dish is still too hot (too cold)" may be immediately presented.

(S111) It is determined whether the food has been eaten completely (whether the amount of all food and drink has reached zero) and whether intake has ended (whether no intake behavior (e.g., chewing) has been performed for a predetermined period of time).
Here, if it is determined that either or both of "eaten the whole meal" and "intake is completed", the process proceeds to step S112 to finally calculate the QoF index.

(S112) On the other hand, if it is determined in step S111 that both "the meal has not been finished" and "the intake has not been completed", it is then determined whether there has been a change in the number of types (number of items) of food and drink that have been served.
If it is determined that "there is a change in the number of types (items) but the number of types (items) is 1 or more" (for example, because a new type of food or drink has been added or one or more types of food or drink have been lowered), the process proceeds to step S101, which is the first step of the eating behavior monitoring method. If it is determined that "there is no change in the number of types (items)", the process proceeds to step S104 to update the food and drink temperature information.

(S113) Furthermore, if it is determined that the "number of types (items) is zero" (for example, because all food and drink items have been removed), the QoF index is calculated and presented based on the information obtained up to this point, and further management is performed, and this eating behavior monitoring method is then terminated.

Returning to the functional block diagram of FIG. 1, the notification means control information generating unit 115b of the feedback unit 115 may generate control information for controlling the irradiation of light onto the specified type of food or drink (soup) by a pre-installed light irradiation means (projector 4 in FIG. 1) capable of irradiating light onto the specified type of food or drink (soup in FIG. 1) based on the "information relating to the timing, time and/or order in which a specified type of food or drink should be consumed" selected or generated as feedback information, and output or transmit this control information to the light irradiation means (projector 4).

As a result, for example, when feedback information indicating that "soup should be consumed first" is selected or generated, the notification means control information generating unit 115b can generate control information indicating that "a projection mapping image with content recommending that the soup be consumed first is projected onto the soup that should be consumed first, and this projection is continued until the consumption behavior of the soup is detected," and send this to the projector 4, causing the projector 4 to carry out the operation of that content.

In this embodiment, the projector 4 is installed and adjusted so that the entire area of the dining table surface is a projectable area, with the dining table surface regarded as a screen. In addition, in order to map and project the feedback information to a position on the specified dining table surface, the notification means control information generating unit 115b generates control information including an image to be projected as feedback information and its projection position, transmits it to the projector 4, and projects the image at the specified projection position. Here, in order to present the feedback information in a form that is easy for diners to understand or accept, the information to be projected may be content composed of animation images, text information, etc. As a preferred modification, the projector 4 may be one that emits a simple spotlight, but on the other hand, it may be equipped with a servo motor that makes it possible to change the irradiation position of the spotlight, and an irradiation position control unit that controls the drive of this servo motor so that the specified position on the dining table surface can be irradiated. In this case, first,
(a) The food and drink recognition unit 112 determines the position information of the soup using the image information (including the served soup) generated by the visible light camera 2;
(b) the notification means control information generating unit 115b generates control information including an instruction to irradiate the position related to the determined soup position information, and transmits the control information to the irradiation position control unit of the projector 4;
(c) The irradiation position control section of the projector 4 drives the servo motor so that the projector 4 can irradiate the position specified (in the received control information).

In this embodiment, the notification means control information generating unit 115b of the feedback unit 115 performs the following based on the "information related to the timing, time, and/or order in which a predetermined type of food or drink should be consumed" selected or generated as feedback information:
(a) a light source that is pre-installed on the utensils used by the diner to eat (in FIG. 1, a notification light is installed on the chopsticks 5 with a notification function);
(b) a vibrator that is pre-installed on the utensils used by the diner to eat (in FIG. 1, the notification vibrator is installed on the chopsticks 5 with notification function);
(c) a light source pre-installed on the diner's food container; and/or
(d) Control information may be generated to control the notification operation to the diner by a vibrator pre-installed on the dishware used by the diner to eat, and this control information may be transmitted to the light source and/or the vibrator.

As a result, for example, when feedback information is selected or generated to the effect that "the food or drink (currently being held between chopsticks) is too hot (its temperature is above the appropriate temperature range)," the notification means control information generating unit 115b can generate control information that generates a vibration to warn that the food or drink is "too hot," and send this to the chopsticks with notification function 5 (which are holding the very hot food or drink), causing the vibrator of the chopsticks with notification function 5 to perform the corresponding operation.

In addition, the notification means control information generating unit 115b generates appropriate control information that matches the feedback information and transmits it to the notification means as described above, making it possible to implement various notifications that reflect the feedback information. For example, for food or drink that needs to be cooled down (is too hot), it is possible to project a projection light onto (a position adjacent to) the food or drink, capable of showing a countdown timer or progress bar indicating the remaining time until the "time when the food or drink may be consumed (when the temperature of the food or drink falls within the appropriate temperature range)" determined from the predicted temperature information.

In addition, when the light lamp mounted on the chopsticks 5 with notification function is capable of emitting light of various colors, the chopsticks 5 with notification function,
(a) When hot food or drink that is too hot (its temperature is above the recommended temperature range) is inserted, the light will immediately turn red,
(b) When hot food or drink within the appropriate temperature range is placed between the two, the sensor will glow green.
(c) When food or drink at room temperature (the same temperature as the ambient temperature) is inserted, the lamp is not turned on.
(d) When cold food or drink that is too cold (its temperature is below the optimum temperature range) is placed between the two, it immediately glows purple,
(e) It is also preferable to display colors that allow the user to see at a glance the relationship between the temperature of the food or drink being consumed and its appropriate temperature range, such as by having the light turn cyan when cold food or drink within the appropriate temperature range is placed between the food or drink and the user's hand.

Also, the spotlight may continue to be shone on foods and drinks whose temperatures are within the optimum temperature range, and the spotlight may be stopped when the temperature goes outside the optimum temperature range. Furthermore, a group of foods and drinks that are determined to go well together may be shone with a spotlight of the same color, for example white, while foods and drinks that should be eaten separately may be shone with spotlights of different colors, for example white and red, respectively. It is also possible to shine a stronger spotlight on foods and drinks that should be eaten next based on their food combinations, to indicate the next candidate for ingestion.

Furthermore, if the food or drink (currently being held between chopsticks) is too hot (its temperature exceeds the optimum temperature range), the speaker 109 of the eating behavior monitor 1 (or an external speaker) may output a voice message such as "That stew is too hot. Please be careful." Also, for food or drink such as "meat buns" whose overall temperature is difficult to estimate from the surface temperature (obtained from an infrared image), it is preferable to confirm that the menu/ingredients optimum temperature DB of the optimum food and drink temperature DB 105 has registered the temperature as "difficult to predict," and then notify the user of this by voice, for example, from the speaker 109.

Also in the functional block diagram of FIG. 1, the food and beverage temperature control information generating unit 115c of the feedback unit 115 may generate control information for controlling the operation of cooling and/or heating the food and beverage by a cooling device and/or heating device pre-installed on the tableware related to the diner's meal, based on the "information related to the timing, time and/or order in which a specified type of food and beverage should be consumed" selected or generated as feedback information, and transmit this control information to the cooling device and/or heating device.

As a result, for example, when feedback information is selected or generated to the effect that "the food or drink (on which chopsticks have just been placed) is too hot (its temperature is above the appropriate temperature range)," the food or drink temperature control information generating unit 115c may generate control information to perform a cooling operation and send this to Peltier element coolers (as multiple cooling devices arranged in a grid pattern inside the surface of the plate) mounted on the plate (on which the food or drink is served), causing these Peltier element coolers to perform an operation to cool the food or drink (to lower the temperature of the food or drink to bring it within the appropriate temperature range).

Here, the heating device may be, for example, an electric heater or an IH heater capable of electromagnetic induction heating. In this case, the food and beverage temperature control information generating unit 115c can transmit control information to an IH heater built into a plate on which food or beverage that is too cold is placed, to perform a heating operation, thereby causing the IH heater to perform an operation of heating the food or beverage (raising the temperature of the food or beverage to within the appropriate temperature range).

As described above in detail, according to the present invention, food and drink temperature information about the food and drink can be obtained not just by itself but also in combination with food and drink content information about the food and drink, making it possible to select or generate feedback information that takes both into consideration. In other words, it is possible to ascertain the temperature of the provided food and drink according to its content, and provide more accurate feedback to the diner.

In addition, by providing such feedback information, it may be possible to encourage diners to change their eating habits to make their food more palatable, or to encourage them to change their eating habits to healthier ones.

Furthermore, in order to encourage diners, including children and the elderly, to adopt healthier eating behavior, it is possible to present feedback information generated by the present invention that takes into account the temperature and contents of the food and drink provided and that is beneficial to health. In other words, the present invention can also contribute to Goal 3 of the United Nations-led Sustainable Development Goals (SDGs) "Ensure healthy lives for all at all ages and promote well-being and end poverty in all its forms everywhere."

Furthermore, for example, in order to encourage diners in each household to adopt eating habits that eliminate problems such as food waste and nutritional imbalance, it is possible to present feedback information generated by the present invention that takes into account the temperature and contents of the food and drink provided and is useful in solving the problems. In other words, the present invention can also contribute to Goal 12 of the United Nations-led Sustainable Development Goals (SDGs), "Ensure sustainable consumption and production patterns."

With respect to the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical ideas and viewpoints of the present invention can be easily made by a person skilled in the art. The above description is merely an example and is not intended to be restrictive in any way. The present invention is limited only by the scope of the claims and their equivalents.

REFERENCE SIGNS LIST 1 Eating behavior monitoring device 101 Communication interface 102 Food and drink temperature information storage unit 103 Food and drink image information storage unit 104 Eating behavior related information storage unit 105 Food and drink optimum temperature database (DB)
106 Food and Drink Combination Database
107 Touch panel display (TP/DP)
108 Microphone 109 Speaker 111 Food and drink/ambient temperature acquisition unit 112 Food and drink recognition unit 113 Food and drink content acquisition unit 114 Eating behavior information generation unit 114a QoF calculation unit 115 Feedback unit 115a Food and drink temperature prediction unit 115b Notification means control information generation unit 115c Food and drink temperature control information generation unit 121 Communication control unit 122 Input/output control unit 2 Visible light camera 3 Thermography camera 4 Projector 5 Chopsticks with ventilation function 6 Electromyography measuring device

Claims (14)

a food and drink temperature acquisition means for acquiring food and drink temperature information, which is information relating to the temperature of the food and drink, from a temperature measurement means capable of measuring the temperature of the food and drink involved in a meal of a diner;
a food and drink content acquisition means for acquiring food and drink content information, which is information related to at least the type of the food and drink for the diner, from a recognition means capable of recognizing at least the type of the food and drink and the amount of the food and drink;
eating behavior information generating means for receiving eating behavior related information from a storage means storing eating behavior related information that is information related to the past eating behavior of the diner, the information including at least information related to the temperature of the food or drink of the type when the food or drink was ingested, and generating eating behavior information by compiling the eating behavior related information;
an eating behavior monitor device comprising: a feedback means for predicting information relating to the future temperature of a type of food and beverage based on the food and beverage temperature information and at least the food and beverage content information relating to the type using a preset standard or model for feedback based on the temperature and content of the food and beverage, and for selecting or generating and providing feedback information including information relating to the timing, time and/or order in which the type of food and beverage should be ingested, adjusted to suit the eater, based on the predicted information relating to the future temperature and information relating to the temperature of the type of food and beverage when ingested, which is included in the eating behavior information. The eating behavior information also includes information regarding the time when the food or drink of the type was consumed,
The eating behavior monitoring device of claim 1, characterized in that the feedback means adjusts information regarding the timing, time and/or order in which the type of food or beverage should be ingested, which is included in the feedback information, to suit the diner, based also on information regarding the time when the type of food or beverage was ingested. The eating behavior information also includes information related to the chewing behavior of the diner when the diner ingests the food or drink of the type, the information being generated from an output of a pre-installed electromyographic sensor capable of detecting the chewing of the diner,
The eating behavior monitoring device of claim 1 or 2, characterized in that the feedback means adjusts information regarding the timing, time and/or order in which the type of food or beverage should be ingested, contained in the feedback information, to suit the diner, based also on information regarding the chewing behavior when the type of food or beverage is ingested. the eating behavior information includes an index of eating behavior quality for the diner, the index including an index relating to the appropriate temperature of the food and drink provided, an index relating to the degree to which the diner finished ingesting the food and drink provided while it was at an appropriate temperature, an index relating to the degree to which the diner finished the meal while the food and drink provided was at an appropriate temperature, an index relating to the number of types of food and drink provided, and/or an index relating to the appropriate temperature of the diner's eating environment;
An eating behavior monitoring device as described in any one of claims 1 to 3, characterized in that the feedback means adjusts information regarding the timing, time and/or order in which the type of food and drink should be ingested, contained in the feedback information, to suit the diner , based also on the quality index of the eating behavior. a food and drink temperature acquisition means for acquiring food and drink temperature information, which is information relating to the temperature of the food and drink, from a temperature measurement means capable of measuring the temperature of the food and drink involved in a meal of a diner;
a food and drink content acquisition means for acquiring food and drink content information, which is information related to at least the type of the food and drink for the diner, from a recognition means capable of recognizing at least the type of the food and drink and the amount of the food and drink;
An eating behavior monitor device characterized by having a feedback means for predicting information regarding the future temperature of a type of food and beverage based on the food temperature information and at least the food and beverage content information related to the type using a preset feedback standard or model based on the temperature and content of the food and beverage, selecting or generating feedback information including information regarding the timing, time and/or order in which the type of food and beverage should be ingested based on the predicted information regarding the future temperature, generating control information for controlling the operation of cooling and/or heating the food and beverage by a cooling device and/or heating device pre-installed on the tableware related to the meal of the diner based on the information regarding the timing, time and/or order in which the type of food and beverage should be ingested, and transmitting the control information to the cooling device and/or the heating device. The food temperature acquisition means also acquires environmental temperature information, which is information related to the temperature of the dining environment of the diner, from the temperature measurement means or another temperature measurement means;
6. The eating behavior monitor according to claim 1, wherein the feedback means predicts information relating to a future temperature of the food or drink of the type in question based on the environmental temperature information as well. The eating behavior monitor device according to any one of claims 1 to 6, characterized in that the temperature measurement means is a thermograph installed in the eating environment of the diner, a thermometer mounted on utensils used by the diner, and/or a thermometer mounted on tableware related to the diner's meal. The eating behavior monitor device according to any one of claims 1 to 7, characterized in that the feedback means generates control information for controlling the irradiation of light onto the food or drink of the selected or generated type by a pre-installed light irradiation means capable of irradiating the food or drink with light, based on information relating to the timing, time and/or order in which the food or drink of the selected or generated type should be ingested, and outputs or transmits the control information to the light irradiation means. The eating behavior monitor device according to any one of claims 1 to 8, characterized in that the feedback means generates control information for controlling a notification operation to the diner by a light source pre-installed on a tool used by the diner for a meal, a vibrator pre-installed on the tool, a light source pre-installed on tableware related to the diner's meal, and/or a vibrator pre-installed on the tableware, based on the selected or generated information related to the timing, time and/or order in which the type of food or drink should be consumed, and transmits the control information to the light source and/or the vibrator. The eating behavior monitor device according to any one of claims 1 to 9, characterized in that the recognition means acquires information relating to an image including the food or drink from a pre-installed camera capable of photographing the food or drink, and recognizes at least the type of the food or drink, among the type and amount, based on the information relating to the image. The eating behavior monitor according to claim 10, characterized in that the eating behavior monitor is an information processing device equipped with the recognition means, capable of receiving information related to the image from the external camera, a tablet computer equipped with the camera and the recognition means, a smartphone equipped with the camera and the recognition means, or an augmented reality (AR) type smart glass equipped with the camera and the recognition means. A temperature measuring means capable of measuring the temperature of food and drink consumed by a diner;
a food and drink temperature acquisition means for acquiring food and drink temperature information, which is information related to the temperature of the food and drink, from the temperature measurement means;
A recognition means capable of recognizing at least the type of the food and drink among the type and amount of the food and drink;
a food and drink content acquisition means for acquiring food and drink content information, which is information related to at least the type of the food and drink of the diner, from the recognition means;
eating behavior information generating means for receiving eating behavior related information from a storage means storing eating behavior related information that is information related to the past eating behavior of the diner, the information including at least information related to the temperature of the food or drink of the type when the food or drink was ingested, and generating eating behavior information by compiling the eating behavior related information;
an eating behavior monitoring system comprising: a feedback means for predicting information relating to the future temperature of a type of food and beverage based on the food and beverage temperature information and at least the food and beverage content information relating to the type using a preset feedback standard or model based on the temperature and content of the food and beverage; and selecting or generating and providing feedback information including information relating to the timing, time and/or order in which the type of food and beverage should be ingested, adjusted to suit the eater, based on the predicted information relating to the future temperature and information relating to the temperature of the type of food and beverage when ingested, which is included in the eating behavior information. A program for causing a computer to function to monitor a person's eating behavior,
a food and drink temperature acquisition means for acquiring food and drink temperature information, which is information relating to the temperature of the food and drink, from a temperature measurement means capable of measuring the temperature of the food and drink involved in a meal of a diner;
a food and drink content acquisition means for acquiring food and drink content information, which is information related to at least the type of the food and drink for the diner, from a recognition means capable of recognizing at least the type of the food and drink and the amount of the food and drink;
eating behavior information generating means for receiving eating behavior related information from a storage means storing eating behavior related information that is information related to the past eating behavior of the diner, the information including at least information related to the temperature of the food or drink of the type when the food or drink was ingested, and generating eating behavior information by compiling the eating behavior related information;
An eating behavior monitor program characterized by having a computer function as a feedback means for predicting information regarding the future temperature of a type of food or drink based on the food and drink temperature information and at least the food and drink content information relating to the type, using a preset standard or model for feedback based on the temperature and content of the food and drink, and selecting or generating and providing feedback information including information regarding the timing, time and/or order in which the type of food or drink should be ingested, adjusted to suit the eater, based on the predicted information regarding the future temperature and information regarding the temperature of the type of food or drink when ingested, included in the eating behavior information. 1. A method for monitoring eating behavior in a computer for monitoring an eating behavior of a person eating a meal, comprising:
a step of acquiring food and drink temperature information, which is information relating to the temperature of the food and drink, from a temperature measuring means capable of measuring the temperature of the food and drink involved in the diner's meal, acquiring food and drink content information, which is information relating to at least the type of the food and drink, of the diner, from a recognition means capable of recognizing at least the type and amount of the food and drink, receiving eating behavior related information, which is information relating to the diner's past eating behavior, from a storage means that stores eating behavior related information including at least information relating to the temperature of the food and drink of that type when it was ingested, and generating eating behavior information that compiles the eating behavior related information;
A method for monitoring eating behavior, comprising the steps of: using a preset standard or model for feedback based on the temperature and contents of the food and beverages, predicting information related to the future temperature of the type of food and beverages based on the food temperature information and at least the food and beverage content information related to the type, and selecting or generating and providing feedback information including information related to the timing, time and/or order in which the type of food and beverages should be ingested, which is adjusted to suit the eater, based on the predicted information related to the future temperature and information related to the temperature of the type of food and beverages when ingested, which is included in the eating behavior information.

Images