JP6480676B2 - Intake amount estimation device, intake amount estimation method and program - Google Patents

Description

  The present invention relates to an intake amount estimation apparatus, an intake amount estimation method, and a program for estimating a user's intake amount using image recognition technology.

In recent years, the number of people who keep meal records is increasing due to the growing health consciousness. Along with this, attempts have been made to estimate food intake.
As the estimation of the amount of food intake, for example, a method for estimating the amount of heat consumed using spectroscopic analysis (Patent Document 1), or a food that is taken in advance by taking an image of a food and eating it using an image recognition process A system (Non-Patent Document 1) that estimates the total amount of heat has been proposed.

In patent document 1, a user's mouth is image | photographed with an income type camera etc., a visible light image and an infrared absorption spectrum are acquired, and a user's intake calorie | heat amount is acquired using the acquired visible light image and an infrared absorption spectrum. An ingestion calorie estimation device has been proposed.
Non-Patent Document 1 proposes a system that takes an image of food before eating and estimates the amount of heat of the food.

JP 2013-36907 A

Y. Kawano and K. Yanai. “Real-time mobile food recognition system.” In Proc. Of CVPR International Workshop on Mobile Vision (IWMV), 2013

However, the method described in Patent Document 1 described above requires special cameras and equipment for spectroscopic analysis and cannot be used easily by the user.
In addition, the system of Non-Patent Document 1 described above can be used for meals whose intake is known in advance, such as set meals, but for meals whose intake is not known in advance, such as pots and yakiniku. It was not available.

  The present invention has been made in view of the above circumstances, and it is not necessary to use a special camera or equipment, and it is possible to estimate the actual intake amount of the user without knowing the intake amount in advance. Provided is an intake estimation device, an intake estimation method, and a program.

  The present invention relates to a detection unit that detects food conveyed to a user's oral cavity as a food region in an image acquired by imaging the user's oral cavity in time series, and a food region detected by the detection unit. Recognized by the food recognition unit, the food recognition unit that performs image recognition processing, recognizes the type of food, the specific amount storage unit that associates and stores the type of food and the specific amount per unit amount of the food type, and the food recognition unit An intake amount estimation unit that acquires a specific amount per unit amount corresponding to the type of food from the specific amount storage unit, and estimates a specific amount of food carried to the oral cavity based on the acquired specific amount as a user's intake amount An ingestion amount estimation device is provided.

  Further, the specific amount per unit amount of the type of food is preferably the heat amount per unit amount of food or the price per unit amount of food.

  The detection unit detects the oral cavity and the user's hand or tableware based on the acquired image, and detects that the oral cavity and the hand or tableware are close to each other by a predetermined distance. It is preferable to detect as a food area.

  In addition, it is preferable to further include an intake amount recording unit that sequentially records the intake amount estimated by the intake amount estimation unit, and the total of the intake amounts sequentially recorded in the intake amount recording unit is preferably used as the meal amount of the user.

In addition, a food amount estimation unit for estimating the size of the food conveyed to the oral cavity from the size of the food region detected by the detection unit is further provided, and the intake amount estimation unit is recognized by the food recognition unit and stores a specific amount It is preferable to estimate the intake based on the specific amount per unit amount corresponding to the type of food obtained from the unit and the size of the food estimated by the food amount estimation unit.

  It is preferable to further include an imaging unit that acquires an image, and a warning unit that issues a warning when the user's face or mouth is not captured for a certain period of time in the image acquired by the imaging unit.

  Further, the present invention provides a detection step of detecting food conveyed to the user's oral cavity as a food area in an image acquired by imaging the user's oral cavity in time series, and the detected food area. A food recognition step for recognizing the type of food, a specific amount per unit amount corresponding to the type of food recognized in the food recognition step, the type of food and the food The specific amount per unit amount is acquired from the specific amount storage unit that stores the corresponding amount, and the specific amount of the food carried to the oral cavity based on the acquired specific amount is used as the intake amount of the user An intake amount estimation method comprising: an intake amount estimation step for estimation is provided.

  Moreover, this invention provides the program for making a computer perform each step of the above-mentioned intake estimation method.

  According to the present invention, during a meal, a user can estimate the actual intake amount without using a special camera or equipment and without performing an input operation.

It is a block diagram which shows the whole structure of the intake amount estimation apparatus which concerns on Embodiment 1 of this invention. It is a flowchart which shows operation | movement of the intake amount estimation apparatus of FIG. It is a figure which shows an example of the display screen of the intake amount estimation apparatus of FIG. (A)-(D) are the figures explaining operation | movement of the detection part of an intake estimation apparatus. It is a block diagram which shows the whole structure of the intake amount estimation apparatus concerning Embodiment 2 of this invention. It is a figure which shows an example of the warning display in the display screen of the intake amount estimation apparatus of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

Embodiment 1
FIG. 1 is a block diagram showing an overall configuration of an intake amount estimation apparatus 10 according to Embodiment 1 of the present invention.
As shown in FIG. 1, the intake amount estimation device 10 includes a detection unit 14, a food recognition unit 16 and an intake amount estimation unit 18 that are sequentially connected from the imaging unit 12, a control unit 20 that is connected to each of these units, and a control unit. A specific amount storage unit 22, a food type storage unit 24, an intake amount recording unit 26, a display unit 28, and an operation unit 30 connected to the unit 20. The control unit 20 includes an internal memory 32 therein.

  The imaging unit 12 includes an imaging optical system such as an imaging lens (not shown) and an imaging element such as a CCD or a CMOS (not shown), and outputs light received by the imaging element through the imaging optical system as captured image data. The captured image data is, for example, moving image data composed of a plurality of still image frames.

  The detection unit 14 performs various detection processes on the captured image based on the captured image data to detect an object appearing in the captured image such as a face, food, and chopsticks as a predetermined image area. Details of the detection process described above in the detection unit 14 will be described later.

  The food recognition unit 16 recognizes the type of food from the image area (hereinafter referred to as food area) of the food detected by the predetermined process, and details of the recognition process will be described later.

  The intake amount estimation unit 18 is based on the recognized food type and the specific amount per unit amount corresponding to the type of food stored in the specific amount storage unit 22 as a database. Is estimated.

  The control unit 20 stores an operation program and the like for controlling each unit of the intake amount estimation device 10 and includes an internal memory 32 that stores various data calculated by each unit. Each unit of the intake amount estimation device 10 is controlled based on an operation program or the like stored in the internal memory 32.

  The specific amount storage unit 22 stores a type of food and a specific amount per unit amount corresponding to the type of food as a database, and a unit corresponding to a predetermined food based on an instruction from the control unit 20 A specific amount per unit is output. Note that examples of the specific amount include food calories and prices.

As with the specific amount storage unit 22, the food type storage unit 24 is configured by an external memory, for example, and stores the image feature amount for each type of food obtained from learning data prepared in advance as a database. An image feature amount is output based on an instruction from the control unit 20. Examples of image feature amounts include SIFT (Scale-Invariant Feature Transform) feature amounts and color feature amounts.
The intake amount recording unit 26 sequentially records the user's intake amount estimated by the intake amount estimation unit 18. Moreover, you may memorize | store not only the user's intake acquired in real time but the user's intake in the past meal, for example.

  The display unit 28 is configured by, for example, a liquid crystal display, an organic EL (Electro-Luminescence) display, and the like, and displays an image captured by the imaging unit 12 based on an instruction from the control unit 20, and for example, a food recognition unit The type of food recognized at 16, the amount of food intake estimated by the intake amount estimation unit 18, and the accumulation of the amount of food intake stored in the internal memory 32 are displayed. The past meal amount stored in the meal amount storage unit 26 may be displayed.

  The operation unit 30 includes, for example, a mechanical switch, a button, and the like, and outputs an input signal based on a user input to the control unit 20. In addition, the operation unit 30 may be configured as a touch panel display together with the display unit 28 described above, configured as a touch panel display, and when the user touches on the display screen, the intake amount estimation device 10 is activated, terminated, etc. Various operations can be performed.

  The internal memory 32 is configured by, for example, a RAM (Random Access Memory). The internal memory 32 exists in the control unit 20, records a program for controlling each part of the intake amount estimation device 10, and calls the program in the control unit 20 based on a user instruction to control the device. And stores various data calculated in each part of the intake amount estimation apparatus 10.

Next, based on the flowchart shown in FIG. 2, the example of the display screen shown in FIG. 3, and the explanatory diagrams shown in FIGS. 4A to 4D, the operation of the intake amount estimation apparatus 10 in FIG. (Quantity estimation method) will be described.
First, as step S1, when the user performs a meal start operation via the operation unit 20 of the intake amount estimation apparatus 10 and the user starts shooting a meal landscape, the image pickup unit 12 captures captured image data of the meal landscape. Is output. The output captured image data is output to the detection unit 14 and is also output to the display unit 28 through the control unit 20.

  FIG. 3 is a diagram illustrating an example of a display screen of the display unit 28. As shown in FIG. 3, the display screen V includes an image display area P1, a food type display area P2, a food count area P3, and a total intake amount display area P4. The image display area P1 is an area for displaying a picked-up image based on the picked-up image data, and the food type display area P2 is an area for displaying the type of food recognized by an image recognition process described later, and the food count area P3. Is an area for counting the number of foods recognized by the image recognition process described later, and the total intake display area P4 is a total of food intakes counted in the food count area P3 (hereinafter referred to as total intake). .) Is an area for displaying.

  Next, as step S <b> 2, the detection unit 14 of the intake amount estimation device 10 detects mouth detection processing for detecting the user's mouth with respect to the captured image based on the captured image data acquired from the imaging unit 12, and detects the user's chopsticks. The chopsticks detection process is performed. In the mouth detection process, for example, the user's mouth region D1 is detected from the captured image as shown in FIG. 4A by using a cascade classifier that has learned the mouth image in advance. Various known methods can be used for the mouth detection process. In the present embodiment, the accuracy of mouth detection is improved by detecting the user's face area in advance and performing mouth detection processing on the lower half of the detected face area.

Further, in the chopsticks detection process, for example, a chopstick region D2 is detected from the captured image as shown in FIG. 4B by extracting a line segment from the captured image by probabilistic Hough transformation. In this embodiment, the line segment extracted from the captured image is set as chopsticks and the periphery of the chopstick is set as the chopstick area D2. However, if the Hough conversion is simply performed, even the background line segment is detected as chopsticks. Therefore, the accuracy of chopsticks detection is improved by extracting a portion that moves between captured images as a front portion and extracting a line segment in the front portion.

  In the subsequent step S3, the detection unit 14 calculates the distance between the mouth area D1 and the chopstick area D2 detected in advance as described above, and the calculated distance is predetermined as shown in FIG. 4C. It is detected that the value is equal to or less than the value or that the chopstick area D2 overlaps the mouth area D1. If the approach between the mouth area D1 and the chopstick area D2 cannot be detected, the process returns to immediately before step S2 to detect the mouth area D1 and the chopstick area D2 again, and the mouth area D1 and the chopstick area D2 again. If the approach to can be detected, the process proceeds to step S4.

  As subsequent step S4, the detection unit 14 detects the food conveyed to the user's mouth as the food region D3 as shown in FIG. 4D, and cuts out the food region D3. An image of the cut food (hereinafter referred to as a food image) is output to the food recognition unit 16.

  In step S5, the food recognition unit 16 performs image recognition processing on the food image output from the detection unit 14, and recognizes the type of food that appears in the food region of the captured image. Various types of known methods can be used to recognize the type of food in the food recognition unit 16. For example, the food recognition unit 16 calculates the image feature amount of the food image in the food area. The food recognizing unit 16 accesses the food type storage unit 24 through the control unit 20 based on the calculated image feature amount, and determines the correspondence between the type of food stored in the food type storage unit 24 and the image feature amount. The type of food (food name) corresponding to the calculated image feature amount is acquired from the recorded database. The acquired type of food is output to the intake amount estimation unit 18 through the control unit 20.

  As step S <b> 6, the intake amount estimation unit 18 accesses the specific amount storage unit 22 through the control unit 20 based on the acquired type of food, and stores the type of food and the unit amount stored in the specific amount storage unit 22. A specific amount per unit amount of food ingested by the user is acquired from a database that records the correspondence with the specific amount. The acquired specific amount is stored in the internal memory 32 of the control unit 20 as the user's intake, and is output to the display unit 28.

The display unit 28 displays the user's meal scenery in real time in the image display area P1, displays the type of food recognized by the food recognition unit 16 in the food type display area P2, and stores it in the internal memory 32 in the food count area P3. The type and number of stored foods are displayed, and the total intake amount of the user at that time stored in the internal memory 32 in the total intake amount display area P4 is displayed.

As step S7, the control part 20 returns to immediately before step S2, and repeats step S2 to step S7 again, when the instruction | indication of a meal end is not input from a user and a user's meal continues. If an instruction to end the meal is input from the user, the process proceeds to step S8.
Note that the control unit 20 may determine that the meal has ended by detecting that the user has left the seat from the captured image or that the chopsticks and the mouth have not been approached for a predetermined time by the detection unit 14.

  Finally, as step S8, the control unit 20 records the total intake amount of the user that has been recorded in the internal memory 32 so far as the meal amount of the user in the meal amount storage unit 26. In addition to the total intake amount, the count number of food, the meal time, and the like may be stored in the meal amount storage unit 26 as a meal amount.

  As described above, according to the intake amount estimation apparatus according to Embodiment 1 of the present invention, the user does not use a special camera or equipment and does not perform input work or the like during a meal. The actual intake can be estimated.

Embodiment 2
FIG. 5 is a block diagram showing the overall configuration of the intake amount estimation apparatus 40 according to Embodiment 2 of the present invention.
As shown in FIG. 5, the intake amount estimation device 40 includes an imaging unit 12, a detection unit 14 that is sequentially connected from the imaging unit 12, a food recognition unit 16, a food amount estimation unit 42, and an intake amount estimation unit 18. A control unit 20 connected to each unit, a specific amount storage unit 22, a food type storage unit 24, a meal record storage unit 26, a display unit 28, and an operation unit 30 connected to the control unit 20, respectively. The control unit 20 includes an internal memory 32 therein. In addition, the imaging part 12, the detection part 14, the food-recognition part 16, the intake amount estimation part 18, the control part 20, the specific amount memory | storage part 22, food type provided with the same structure as the intake amount estimation apparatus 10 which concerns on Embodiment 1. The storage unit 24, the meal record storage unit 26, the display unit 28, the operation unit 30, and the internal memory 32 are assigned the same reference numbers, and the description thereof is omitted.

The food amount estimation unit 42 estimates the amount of food consumed by the user from the area of the food image detected by the detection unit 14 and cut out (area of the food region).
The food amount estimation unit 42 outputs the estimated food amount to the intake amount estimation unit 18, and the intake amount estimation unit 18 estimates the specific amount per unit amount based on the type of food and the food amount estimation unit 42. The user's intake is estimated based on the amount of food. When the amount of food estimated by the food amount estimation unit 42 is large, the user's intake is estimated to be large, and when the estimated amount of food is small, the user's intake is estimated to be small.

  According to the intake amount estimation device according to Embodiment 2 of the present invention, in addition to the effects shown in Embodiment 1, since the food amount estimation unit 42 is provided, the area of the food can be calculated from the food image. In addition, by storing the specific amount stored in the specific amount storage unit 22 in units of grams, it is possible to estimate a more accurate food intake by integrating with the area.

Moreover, the intake amount estimation apparatus of the present invention may include a warning unit that issues a warning when the user deviates from the captured image.
As shown in FIG. 6, when the user's face greatly deviates from the captured image, for example, the detection unit of the intake amount estimation device detects that in the face detection process in the previous stage of mouth area detection, and passes through the control unit. A warning to that effect may be displayed in the warning display area P5 of the display screen V.

The warning unit may not only display a warning on the display screen V, but may give a warning to that effect, for example, by sounding a sound. The warning unit of the present invention includes a control unit and a detection unit.
The intake amount estimation device of the present invention includes a warning unit, so that the user can be notified that the intake amount is not estimated correctly. The user can check the intake display by checking the warning display area P5. More accurate recognition can be performed by notifying that the device is not correctly recognized.

The detection unit, the food recognition unit, the intake amount estimation unit, and the control unit (excluding the internal memory) of the intake amount estimation device of the present invention are a CPU (central processing unit) and a program for causing the CPU to perform various processes. (The program of the present invention), but these may be constituted by a digital circuit. As a recording medium for recording the program, various known recording media such as a hard disk, a flash memory, and a DVD-ROM can be used.
The intake amount estimation apparatus and program of the present invention are incorporated in a portable information terminal such as a mobile phone or a digital camera, and the intake amount estimation method of the present invention is used in these portable information terminals.

  The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10, 40 Intake amount estimation apparatus 12 Imaging part 14 Detection part 16 Food recognition part 18 Intake amount estimation part 20 Control part 22 Specific amount memory | storage part 24 Food seed | species memory | storage part 26 Intake amount memory | storage part 28 Display part 30 Operation part 32 Internal memory 42 Food quantity estimation part P1 Image display area P2 Food type display area P3 Food count area P4 Total intake display area P5 Warning display area D1 Mouth area D2 Chopstick area D3 Food area V Display screen

Claims (8)

In an image acquired by imaging a user's mouth in time series, a detection unit that detects food carried to the user's mouth with chopsticks as a food region;
A food recognition unit that performs image recognition processing on the food area detected by the detection unit and recognizes the type of the food;
A specific amount storage unit that stores a type of food and a specific amount per unit amount of the type of food in association with each other;
A food amount estimation unit for estimating a food amount of the food carried to the mouth from the size of the food region detected by the detection unit;
The specific amount per unit amount corresponding to the type of food recognized by the food recognition unit is acquired from the specific amount storage unit, and the acquired specific amount per unit amount and the food amount estimation unit An intake amount estimation unit for estimating an intake amount of food carried to the user's mouth based on the estimated food amount ;
The detector is
From the image, the mouth area where the user's mouth is photographed is detected, and the chopstick area where the user's chopsticks are photographed is detected,
Distance between the inlet region and the chopsticks region, from the image at the time of equal to or less than a predetermined value, intake estimation device characterized that you detect the food area.   The intake amount estimation apparatus according to claim 1, wherein the specific amount per unit amount of the food type is a heat amount per unit amount of the food or a price per unit amount of the food. An intake amount recording unit for sequentially recording the intake amount estimated by the intake amount estimation unit;
The intake amount estimation apparatus according to claim 1, wherein a total amount of the intake amounts sequentially recorded in the intake amount recording unit is used as the user's meal amount. An imaging unit for acquiring the image;
4. The apparatus according to claim 1, further comprising a warning unit that issues a warning when the user's face or the mouth is not captured for a certain period of time in the image acquired by the imaging unit. The ingestion amount estimation device described.   5. The detection unit according to claim 1, wherein the detection unit detects, as the chopstick region, a periphery of a line segment detected by the probabilistic Hough transform among the moving parts of the image. The intake amount estimation apparatus according to 1.   The detecting unit detects a face area of the user, and detects the face area using a cascade classifier that has learned a mouth image for the lower half of the detected face area. The intake amount estimation apparatus according to any one of claims 1 to 5. In an image acquired by imaging the user's mouth in time series, a detection step of detecting food carried to the user's mouth with chopsticks as a food area;
Image recognition processing is performed on the food area detected in the detection step, and the food recognition step for recognizing the type of food,
A food amount estimation step for estimating a food amount of the food conveyed to the mouth from the size of the food region detected by the detection unit;
The specific amount per unit amount corresponding to the type of food recognized in the food recognition step from the specific amount storage unit that stores the type of food and the specific amount per unit amount of the food type in association with each other And estimating the amount of food carried to the user's mouth based on the acquired specific amount per unit amount and the food amount estimated by the food amount estimation step. ,
The detecting step includes
From the image, the mouth area where the user's mouth is photographed is detected, and the chopstick area where the user's chopsticks are photographed is detected,
Intake estimation method the distance between the outlet region and the chopsticks region, to the image at the time of equal to or less than a predetermined value, characterized that you detect the food area. The program for making a computer perform each step of Claim 7 .