JP5790080B2 - Meal image analysis method, meal image analysis program, and meal image analysis apparatus - Google Patents

Description

  The present invention relates to a meal image analysis method, a meal image analysis program, and a meal image analysis apparatus.

  Conventionally, it is known that a well-balanced meal can be taken by ingesting foods of various colors. Nutritionists etc. provide nutritional guidance such as promoting a balanced diet that includes various colors.

  In addition, there is a technique for measuring the amount of meals ingested by the user by analyzing images of meals ingested by the user for nutrition management. Furthermore, there is a technique for determining whether a group such as a staple food or a side meal is being taken in a balanced manner from a meal image according to a meal balance guide that is one index of nutritional balance.

JP 2008-204105 A

  However, the above-described conventional technique has a problem that it is not easy for the user to grasp the balance of meals. In other words, the technique for measuring the amount of meal consumed by the user measures the amount of intake by the user's meal, but cannot grasp the balance of the meal. In addition, the technology for determining whether a group of staple foods, side dishes, etc. is in good balance can grasp whether a group of staple foods, side dishes, etc. is in good balance, but it is possible to ingest various colors of ingredients. It cannot be determined.

  Therefore, the present invention has been made to solve the above-described problems of the prior art, and an object of the present invention is to make a user easily understand the balance of meals.

In order to solve the above-described problems and achieve the object, this meal image analysis method specifies a plurality of colors included in an image portion of food extracted from a captured image, and a plurality of colors between the specified plurality of colors. The ratio of the area occupied by each of the colors is calculated, and information on the nutritional balance of the food is output based on the calculated ratio .

  The disclosed method allows the user to easily grasp the balance of meals.

FIG. 1 is a block diagram illustrating a functional configuration of the meal image analysis apparatus according to the first embodiment. FIG. 2 is a diagram illustrating an example of a meal image. FIG. 3 is a diagram illustrating an example of the shape of a registered vessel. FIG. 4 is a diagram illustrating a result of specifying the position of the food material. FIG. 5 is a diagram for explaining the vanishing point. FIG. 6 is a diagram for explaining processing for converting the shape of a registered vessel in accordance with a line toward the vanishing point. FIG. 7 is a diagram for explaining the process of extracting food image. FIG. 8 is a diagram for explaining the color frequency distribution of the food image. FIG. 9 is a diagram for explaining the process of estimating the quantity from the number and size of the vessels. FIG. 10 is a diagram illustrating an example of information regarding meal balance. FIG. 11 is a flowchart illustrating the processing operation of the meal image analyzing apparatus according to the first embodiment. FIG. 12 is a diagram illustrating a computer that executes a meal image analysis program.

  Hereinafter, embodiments of a meal image analysis method, a meal image analysis program, and a meal image analysis apparatus according to the present invention will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments.

  In the following examples, the configuration and processing flow of the meal image analyzing apparatus according to the first example will be described in order, and finally the effects of the first example will be described.

[Configuration of meal image analyzer]
First, the functional configuration of the meal image analyzing apparatus will be described with reference to FIG. FIG. 1 is a block diagram illustrating the configuration of the meal image analysis apparatus according to the first embodiment. As shown in FIG. 1, the meal image analysis apparatus 10 includes an input unit 11, an output unit 12, a control unit 13, and a storage unit 14. The processing of each of these units will be described below.

  The input unit 11 inputs a meal image including a meal portion taken by a camera or the like, and includes a keyboard and a mouse. The output unit 12 displays a meal balance value obtained by scoring a meal balance, and includes a monitor, a speaker, and the like.

  The control unit 13 has an internal memory for storing a program that defines various processing procedures and the necessary data, and executes various processes. The control unit 13 includes an acquisition unit 13a, an extraction unit 13b, a creation unit 13c, a color calculation unit 13d, and a score calculation unit 13e. Note that a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like is applied to the control unit 13.

  The acquisition unit 13a acquires a meal image including a meal portion input by the input unit 11. Here, the meal image which the acquisition part 13a acquires is demonstrated using the example of FIG. FIG. 2 is a diagram illustrating an example of a meal image. For example, as illustrated in FIG. 2, the acquisition unit 13 a acquires a meal image in which a meal part such as rice, miso soup, fish, and pickles and a part other than the meal part such as a bowl and chopsticks are shown as a meal image. To do. Thereafter, the acquisition unit 13a notifies the extraction unit 13b of the acquired meal image.

  The extraction unit 13b extracts a meal portion from the meal image acquired by the acquisition unit 13a. For example, the extraction unit 13b detects tableware from the meal image and extracts only the meal portion using the shape feature of the tableware. First, the tableware contained in a meal image is demonstrated using FIG. There are various variations in the shape of the tableware included in the meal image, but many are circular, elliptical, and rectangular. For this reason, as illustrated in FIG. 3, the shape of tableware such as a circle, an ellipse, and a rectangle is registered in advance as a shape to be detected. FIG. 3 is an example of the shape of a registered vessel. In the example of FIG. 3, the first square, the second elliptical, the third rectangular, the fourth circular, etc. are registered.

  Then, the extraction unit 13b detects a circle, an ellipse, a quadrangle, and the like in the meal image by using existing techniques such as geometric hashing and generalized Hough transform. Referring to the example of FIG. 4, the extraction unit 13b detects three circular dishes and one rectangular dish from the meal image. FIG. 4 is a diagram illustrating a result of specifying the position of the food material. Here, geometric hashing is a shape detection method that can cope with translation, rotation, and scale changes. The generalized Hough transform is a technique for detecting an arbitrary geometric figure by combining a shape, a pose, and a rotation angle.

  In the process of the extraction unit 13b, when the meal image is taken from an oblique direction, the shape of the tableware may be deformed to cause inconvenience in detecting the shape of the tableware. For this reason, as shown in FIG. 5, the extraction unit 13b extracts a straight line from the image by Hough transform, and calculates a vanishing point where the parallel lines intersect. FIG. 5 is a diagram for explaining the vanishing point.

  Then, as illustrated in FIG. 6, the extraction unit 13 b uses the calculated vanishing point to transform the shape of the tableware registered in advance (see FIG. 3) into a shape that matches the captured image. And the extraction part 13b can detect the shape of tableware even from the meal image by which the meal diagonal image was image | photographed diagonally by detecting the image of tableware as the shape of the tableware which should detect the deformed shape. . FIG. 6 is a diagram for explaining processing for converting the shape of a registered vessel in accordance with a line toward the vanishing point.

  After detecting the shape of the tableware, the extraction unit 13b extracts an image inside the detected tableware. The detailed processing will be described with reference to FIG. 7. As illustrated in FIG. 7, the extraction unit 13 b creates a mask image in which the outside of the tableware shape is painted black based on the meal image in which the tableware shape is detected. . And the extraction part 13b superimposes a meal image and a mask image, and extracts only a meal part, as shown in FIG. FIG. 7 is a diagram for explaining the process of extracting food image. Thereafter, the extraction unit 13b notifies the data of the extracted meal portion to the creation unit 13c.

  The creation unit 13c creates a color frequency distribution of the meal portion extracted by the extraction unit 13b. For example, the creation unit 13c acquires RGB values from the digital data of the food material image and converts them into information on hue (H), saturation (S), and luminance (I). If the example of a conversion type | formula is given, the preparation part 13c will calculate RGB value to hue (H) using the following (1) Formula. Further, the creating unit 13c calculates the RGB value from the equation (2) to saturation (S). Further, the creation unit 13c converts the RGB value from the luminance (I) using the equation (3).

  Then, as shown in FIG. 8, the creating unit 13c determines the color of each pixel of the meal portion in the meal image, classifies the pixels of the meal portion in the meal image for each color, and calculates the color frequency distribution of the meal portion. create. In the example of FIG. 8, the pixels of the meal portion in the meal image are classified by color to create a color frequency distribution of the meal portion. Thereafter, the creation unit 13c stores the created color frequency distribution in the color information storage unit 14a.

  The color calculation unit 13d calculates the number of color types and the color ratio included in the meal portion based on the color frequency distribution stored in the color information storage unit 14a. For example, in the example of FIG. 8, the colors included in the meal portion are green, yellow, red, brown, white, and black (others), so the color calculation unit 13d calculates the number of color types “6”.

  The color calculation unit 13d calculates a color ratio that is a ratio of the number of pixels of each color to the number of pixels of the meal portion. For example, in the example of FIG. 8, the color calculation unit 13d determines that 12% for green, 8% for yellow, 12% for red, 25% for brown, 4% for white, and 4% for black (others) based on the color frequency distribution. Is calculated to be 39%. Thereafter, the color calculation unit 13d stores the calculated number of color types and color ratio in the color information storage unit 14a.

  The score calculation unit 13e reads the number of color types and the color ratio from the color information storage unit 14a, and calculates a meal balance value based on the number of color types and the color ratio. For example, the score calculation unit 13e calculates the meal balance numerical value so that the meal balance numerical value increases as the number of color types increases and the color ratio approaches the preset color ratio. Here, an example of a method for calculating a meal balance value (hereinafter referred to as a score) will be described in detail. The score calculation unit 13e determines whether the ratio of each color in the meal portion is greater than a predetermined threshold, and calculates the score by multiplying the number of colors greater than the threshold by the color ratio greater than the threshold. Thereafter, the score calculation unit 13e stores the calculated score in the meal balance storage unit 14b. Note that the score calculation unit 13e may calculate the score based only on the color type.

  For example, when the threshold is set at 10% and the ratio of each color in the meal portion is 60% white, 24% green, 8% red, 8% yellow, the score calculation unit 13e Determines that two colors, white and green, are larger than the threshold. Then, the score calculation unit 13e calculates that the score is “1.68” by “2 colors × (0.6 + 0.24) = 1.68”. This score indicates that the higher the value, the better the meal balance. For example, when one color is 100%, that is, when the balance of the meal is bad, “1 color × 1.0 = 1.0”, the score is “1.0”, and the score is the above example. It becomes low compared with.

  Further, the score calculation unit 13e may increase or decrease the score according to the number of meals, the meal time zone, the amount of meal, the meal location, etc., as well as the color balance in the meal part. Hereinafter, a method for calculating a score other than the color balance of the meal will be described. For example, the score calculation unit 13e adds 3 points to the score if the number of meals per day is 3 times, 2 points if the number of meals is 2 times, and 1 point otherwise. Moreover, the score calculation part 13e subtracts 0.5 points from a score, when there is a meal for 3 hours before going to bed. To give a specific example, if three colors a day, one of which was before going to bed, 2.5 points are added to the score by “3 meals−0.5 (subtraction before going to bed) = 2.5”. .

  In addition, for example, the score calculation unit 13e takes a histogram of the color of the meal for one month, and how often the same color scheme (for example, 60% yellow for curry, 40% white, etc.) appears or is the same Check if the color scheme is continuous and increase / decrease points. As a specific example, when there are 10 or more color variations of the meals eaten in one month, 10 points are added. Thereby, the tendency of the unbalanced diet in a 1-month meal can be reflected in a score.

  For example, the score calculation unit 13e increases or decreases the score according to the amount of meal. Here, a method for estimating the amount of meal from a meal image will be described with reference to FIG. FIG. 9 is a diagram for explaining the process of estimating the quantity from the number and size of the vessels. The score calculation unit 13e calculates the amount of meal based on the number of dishes and the size of the dishes. Here, as shown in FIG. 9, the size of the tableware is calculated based on the size of the chopsticks.

  In other words, the size of chopsticks is often the same size compared to tableware and trays, and by calculating the size of chopsticks based on chopsticks that have a smaller variation in size compared to tableware and trays, The size can be estimated appropriately. After calculating the amount of meal, the score calculation unit 13e adds 1 point if the amount of meal is within the predetermined range, and 1 point if the amount of meal is outside the predetermined range because the amount of meal is large or small. Subtract.

  For example, the score calculation unit 13e increases or decreases the score according to the meal location. For example, the score calculation unit 13e acquires location information by GPS (Global Positioning System) or the like, and adds 1 point when eating at home, and subtracts 1 point when eating outside the home.

  As described above, the score calculation unit 13e outputs the score via the output unit 12 after calculating the score. Here, the score calculation unit 13e may calculate and output a score for each meal, calculate and output a score for each day, or integrate the scores for one month. You may calculate and output. Further, as illustrated in FIG. 10, the score calculation unit 13e may create a daily meal table in which information such as meal time, meal amount, color distribution, and location in a day is recorded every hour. . FIG. 10 is a diagram illustrating an example of information regarding meal balance. In this case, when outputting the score, the score calculation unit 13e may also output the meal table for the day.

  The storage unit 14 stores data and programs necessary for various processes performed by the control unit 13. The storage unit 14 includes a color information storage unit 14a and a meal balance storage unit 14b. The storage unit 14 is a semiconductor memory device such as a RAM (Random Access Memory), a ROM (Read Only Memory), or a flash memory, or a storage device such as a hard disk or an optical disk.

  The color information storage unit 14a stores the color frequency distribution created by the creation unit 13c, the number of color types calculated by the color calculation unit 13d, and the color ratio. The meal balance storage unit 14b stores the score calculated by the score calculation unit 13e and the daily meal table (see FIG. 10) created by the score calculation unit 13e.

[Processing by meal image analyzer]
Next, processing performed by the meal image analysis apparatus 10 according to the first embodiment will be described with reference to FIG. FIG. 11 is a flowchart illustrating the processing operation of the meal image analyzing apparatus according to the first embodiment.

  As shown in FIG. 11, first, the acquisition unit 13a of the meal image analysis apparatus 10 acquires a meal image (step S101). And the extraction part 13b of the meal image analysis apparatus 10 detects the position of a bowl from a meal image (step S102), and extracts only a meal part (step S103). For example, the extraction unit 13b detects the shape of the tableware by detecting a circle, an ellipse, a rectangle, or the like in the meal image using the existing techniques such as geometric hashing or generalized Hough transform. An image inside the tableware is extracted as a meal portion.

  Subsequently, the creation unit 13c of the meal image analysis apparatus 10 classifies the pixel values of the food image for each color and creates a color frequency distribution of the food image (step S104). For example, the creation unit 13c acquires RGB values from the digital data of the food material image and converts them into information on hue (H), saturation (S), and luminance (I). Then, the creation unit 13c determines the color of each pixel of the meal portion in the meal image, classifies the pixels of the meal portion in the meal image for each color, and creates a color frequency distribution of the meal portion.

  Then, the color calculation unit 13d of the meal image analysis apparatus 10 calculates the number of color types and the color ratio (step S105). Subsequently, the score calculation unit 13e of the meal image analysis apparatus 10 calculates a score based on the number of color types and the color ratio (step S106). For example, the score calculation unit 13e determines whether the ratio of each color in the meal portion is greater than a predetermined threshold, and calculates the score by multiplying the number of colors greater than the threshold by the color ratio greater than the threshold. Thereafter, the output unit 12 outputs the calculated score (step S107) and ends the process.

[Effect of Example 1]
As described above, the meal image analysis apparatus 10 according to the first embodiment acquires a meal image, extracts a meal portion from the acquired image, and creates a color frequency distribution of the meal portion. Based on the color frequency distribution, the number of color types and the color ratio included in the meal portion are calculated, a score based on the number of color types and the color ratio is calculated, and the score is output. For this reason, it is possible to make a user grasp the balance of a meal easily.

  In addition, the meal image analysis apparatus 10 according to the first embodiment calculates the score so that the meal balance value increases as the number of color types increases and as the color ratio approaches the preset color ratio. For this reason, it is possible to calculate the score which shows a meal balance more accurately.

  In addition, the meal image analysis apparatus 10 according to the first embodiment increases or decreases the score according to the number of meals, the meal time zone, the amount of meal, and the meal place, so that the score indicating the meal balance can be calculated more accurately. Is possible.

  In addition, the meal image analysis apparatus 10 according to the first embodiment calculates and integrates the meal balance values in a predetermined period and outputs the calculated meal balance values in the predetermined period, so that long-term meal balance can be easily performed. It is possible to make the user grasp.

  Now, although the first embodiment has been described so far, various different forms other than the first embodiment described above may be implemented. Therefore, another embodiment included in the present embodiment will be described below as a second embodiment.

(1) System Configuration, etc. Further, each component of each illustrated apparatus is functionally conceptual and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. For example, the acquisition unit 13a and the extraction unit 13b may be integrated.

(2) Program By the way, various processes described in the above embodiments can be realized by executing a program prepared in advance by a computer. In the following, an example of a computer that executes a program having the same function as that of the above embodiment will be described with reference to FIG. FIG. 12 is a diagram illustrating a computer that executes a meal image analysis program.

  As shown in FIG. 12, a computer 600 serving as a meal image analyzing apparatus is configured by connecting an HDD 610, a RAM 620, a ROM 630, and a CPU 640 via a bus 650.

  In the ROM 630, a meal image analysis program that exhibits the same function as in the above embodiment, that is, as shown in FIG. 12, an acquisition program 631, an extraction program 632, a creation program 633, a color calculation program 634, and a score calculation are obtained. A program 635 is stored in advance. In addition, about the programs 631-635, you may integrate or disperse | distribute suitably like each component of the meal image analysis apparatus 10 shown in FIG.

  Then, the CPU 640 reads these programs 631 to 635 from the ROM 630 and executes them, so that each program 631 to 635 has an acquisition process 641, an extraction process 642, a creation process 643, and a color calculation process, as shown in FIG. 644 and a score calculation process 645.

  The HDD 610 is provided with color data 611 and score data 612 as shown in FIG. The CPU 640 registers data for the color data 611 and the score data 612, reads the color data 611 and the score data 612, stores the data in the RAM 620, and executes processing based on the data stored in the RAM 620.

DESCRIPTION OF SYMBOLS 10 Meal image analyzer 11 Input part 12 Output part 13 Control part 13a Acquisition part 13b Extraction part 13c Creation part 13d Color calculation part 13e Score calculation part 14 Storage part 14a Color information storage part 14b Meal balance storage part

Claims (10)

Computer
Identify multiple colors in the food image portion extracted from the captured image,
Calculating the ratio of the area occupied by each of the plurality of colors between the plurality of identified colors;
A meal image analysis method, comprising: outputting information relating to the nutritional balance of the food based on the calculated ratio . Obtaining the captured image stored in the storage area;
Identify the image part of the food included in the acquired captured image,
Based on the value of each of a plurality of pixels included in the identified image portion of the food, a color frequency distribution of the plurality of pixels is created,
Based on the created color frequency distribution, calculate the number of color types included in the image portion of the food,
Based on the calculated number of color types, a meal balance value is calculated,
The meal image analysis method according to claim 1, wherein the calculated meal balance value is output. The computer further includes:
Based on the color frequency distribution, calculate the proportion of colors included in the image portion of the food,
In the process of calculating the meal balance value,
The meal image analysis method according to claim 2, wherein the meal balance value is calculated based on the color ratio.   4. The meal balance value is calculated so that the meal balance value increases as the number of color types increases and the color ratio approaches a preset color ratio. Meal image analysis method.   The meal image analysis method according to any one of claims 2 to 4, wherein the meal balance value is increased or decreased according to the number of meals, a meal time zone, the amount of meals, and a meal place. Calculate and integrate the meal balance values within a given period,
The meal image analysis method according to any one of claims 2 to 5, wherein the meal balance numerical value in the calculated predetermined period is output.   7. The meal balance value is calculated based on a color arrangement tendency of the image portion of the food within the predetermined period based on the color frequency distribution for each meal within the predetermined period. The meal image analysis method as described in one.   The meal image analysis method according to any one of claims 1 to 7, wherein tableware is detected from the captured image, and an image portion inside the detected tableware is specified as an image portion of the food. On the computer,
Identify multiple colors in the food image portion extracted from the captured image,
Calculating the ratio of the area occupied by each of the plurality of colors between the plurality of identified colors;
Based on the calculated ratio , information on the nutritional balance of the food is output.
A meal image analysis program for executing processing. A calculation unit that identifies a plurality of colors included in an image portion of food extracted from the captured image, and calculates a ratio of an area occupied by each of the plurality of colors between the plurality of identified colors ;
An output unit that outputs information on the nutritional balance of the food based on the calculated ratio ;
A meal image analyzing apparatus characterized by comprising:

Images