KR101562364B1 - Automatic calorie caculation method using food image and feeding behavior managing system using thereof - Google Patents

Abstract

The present invention relates to a calorie automatic calculation method using food photographs, the method comprising the steps of: (a) calculating image characteristics of color, texture and shape of a study food photograph; (b) generating an automatic classifier through machine learning based on the computed image feature values; (c) extracting image features of color, texture, and shape from input food photographs and inputting them to the automatic classifier to estimate the type and amount of food; And (d) calorie calculation using the type and amount of the estimated food.
In addition, the present invention relates to a system and method for managing eating habits using an automatic calorie calculation method using food photographs, and includes a user terminal for uploading food photographs before and after food on a web or an app. A database storing calorie data for each type of food; And a food estimating module for extracting image characteristics from food photographs before and after the food is uploaded from the user terminal and inputting the image characteristics into an automatic classifier previously learned about the type and amount of food to estimate the type and quantity of food of the food photographs, A calorie calculation module for calculating an intake calorie using a difference between the estimated food type and a food quantity before and after intake and calorie data of a database and a dietary habit management module for guiding and managing a dietary habit of a user using the calculated intake calorie And a server.
According to the present invention, the type and amount of food are estimated using the image characteristics of the color, shape and texture of the food photograph, and the calories are automatically calculated using the type and amount of the estimated food, And it is convenient because it does not need to input the amount manually. If a user uploads food photographs on the basis of web or app, calorie consumed in real time is computed and based on this, a diet consulting suitable for each user is provided.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an automatic calorie counting method using a food photograph,

The present invention relates to an automatic calorie counting method using food photographs and a eating habit management system using the method. More particularly, the present invention relates to a method and apparatus for estimating the type and amount of food by analyzing image characteristics of food photographs, To a system for calculating the calorie of food and managing the eating habits of the user by using the calculated calories.

As modern society becomes more westernized and industrialized, more and more people are suffering from overweight or obesity. Overweight or obesity is a serious problem that causes genetic diseases such as adult diseases in individuals, but it is one of the serious social problems in modern society because it causes huge social costs due to societies.

Therefore, such a diet to overcome the obesity and overweight is essential to modern people is being carried out widely. In addition, recently, as the desire for beauty of people is increased, dieting is being seen as a means to maintain beautiful body and health, rather than overweight or obesity solution for simple health.

In order to do such a diet, it is necessary to analyze his / her eating habits, and it is basically to recognize the calories ingested by himself / herself.

Korean Patent Laid-Open Publication No. 10-2005-0045190 discloses a conventional calorie calculation method. In the mobile communication terminal, calorie calculation information for each food is provided, and according to a user's numerical input, Provides a way to calculate the caloric content.

More particularly, the present invention relates to a method for calorie counting, comprising: a first step of providing a screen displaying calorie calculation information for each food when a user selects a calorie calculation menu on the mobile communication terminal; A second step of calculating the total calories according to the second step, and a third step of displaying the calculated total calories amount.

The conventional method as described above has an effect that the calories can be calculated anytime and anywhere and the calories can be calculated relatively simply by inputting simple numerical values.

However, in the case of the above conventional calorie counting method, since the user still has to input the amount consumed after searching for the type of the food he / she has ingested, the inconvenience due to the manual input and the calculated calorie There is a problem in that it is inaccurate.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and it is an object of the present invention to provide a method and apparatus for calorie counting, in which, without inputting the type and amount of food consumed directly by a user, And a method of automatically calculating calories based on the type and amount of food extracted by extracting features, and a food management system using the method.

According to another aspect of the present invention, there is provided a method for calorie counting using a food photograph, including: a user terminal for uploading food photographs before and after food intake on a web or an app; A method for automatically calculating a calorie using a food photograph using a dietary management system including a stored database and a server having a food estimation module, a calorie calculation module and a dietary management module, the server of the dietary management system comprising: Calculating image features for colors, textures and shapes of the learning food pictures; (b) generating an automatic classifier through machine learning based on the computed image feature values; (c) extracting image features of color, texture, and shape from input food photographs and inputting them to the automatic classifier to estimate the type and amount of food; And (d) calorie calculation using the type and amount of the estimated food; Wherein the color image feature is an RGB color histogram, the shape image feature is a number of pixels and a run length, and the texture image feature includes a gradient histogram and a gray-level co-occurrence matrix (GLCM) And is a texture variable.

In order to accomplish the above object, the present invention provides a system and method for automatically counting calories using food photographs, the system comprising: a user terminal for uploading food images before and after food intake on a web or an app; A database storing calorie data for each type of food; And a food estimating module for extracting image characteristics from food photographs before and after the food is uploaded from the user terminal and inputting the image characteristics into an automatic classifier previously learned about the type and amount of food to estimate the type and quantity of food of the food photographs, A calorie calculation module for calculating an intake calorie using a difference between the estimated food type and a food quantity before and after intake and calorie data of a database and a dietary habit management module for guiding and managing a dietary habit of a user using the calculated intake calorie Wherein the food estimating module comprises: an automatic classifier for mechanically learning the type and amount of food based on image characteristics of colors, textures, and shapes extracted from the pictures of the food for learning; An image feature extraction unit for extracting an image feature of the shape, Wherein the color image feature is an RGB color histogram, and the shape image characteristic includes a number of pixels and a run length of the image ), And the texture image characteristic is a texture parameter of a gradient histogram and a gray-level co-occurrence matrix (GLCM).

According to the present invention, the type and amount of food are estimated using the image characteristics of the color, shape and texture of the food photograph, and the calories are automatically calculated using the type and amount of the estimated food, And it is convenient because it does not need to input the amount manually. If a user uploads food photographs on the basis of web or app, calorie consumed in real time is computed and based on this, a diet consulting suitable for each user is provided.

1 is a flowchart illustrating an automatic calorie calculation method using a food photograph according to the present invention.
2 is a flowchart illustrating an automatic classifier generating process according to the present invention.
3 is a reference view for explaining a method of preprocessing and dividing a food photograph according to the present invention.
4 is a reference diagram for explaining a method of estimating the kind of food using the automatic classifier of the present invention.
5 is a reference diagram illustrating a method of estimating the amount of food using the automatic classifier of the present invention.
6 is a diagram showing calorific data of a database used in the present invention.
Figure 7 is a graphical representation of calorie counts in a food photograph using the method according to the present invention.
FIG. 8 is a graphical representation of calorie intake calculated using a method according to the present invention and displayed on a food photograph. FIG.
FIG. 9 is an exemplary table in which image features of rice (white rice) according to the present invention are extracted.
FIG. 10 is a diagram showing an example of the image characteristics of the kimchi according to the present invention.
11 is a block diagram illustrating a dietary management system using the method of the present invention.
12 is a block diagram illustrating a configuration of a food estimation module according to the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily carry out the technical idea of the present invention.

FIG. 1 is a flowchart illustrating an automatic calorie calculation method using a food photograph according to the present invention. FIG. 2 is a flowchart illustrating an automatic classifier generating process according to the present invention. A reference diagram for explaining a method of automatically calculating a calorie using a photograph. 9 and 10 are tables showing image feature extraction values for kimchi and rice when each pixel of a food photograph expresses RGB 8-bit color.

Referring to FIG. 1, an automatic calorie counting method using food photos according to the present invention includes generating an automatic classifier S100, estimating the type and amount of food using the automatic classifier S200, (S200). The step S200 includes steps of pre-processing the input food image (S210), dividing the pre-edited food image (S220), and calculating the calories using the type and amount of the food. A step S230 of extracting image features by image processing of the divided regions, and a step S240 of inputting the extracted image features to the automatic classifier to estimate the type and amount of the food.

The step S100 of generating the classifier includes a step S110 of obtaining a learning food picture as shown in FIG. 2, a step S120 of pre-processing the obtained learning food picture, A step S140 of obtaining an image feature from the divided learning food photographs, and a step S150 of generating an automatic classifier by inputting the type and amount of food for the extracted image feature.

The step S110 is a step of acquiring a photograph of a food for learning to generate an automatic classifier for the type and amount of the food, wherein the photograph of the food for learning is a photograph of a plate with food or a photograph of a table on which food is placed. The food photograph for learning can be an image photographed using a digital camera such as a camera of a smart phone or a DSLR.

The step S120 is a step of preprocessing the food photograph to easily extract the image feature from the food photograph for learning. The food photograph is reduced or enlarged to a predetermined resolution, for example, 1024 x 768, And recognizing the boundary.

On the other hand, Line Hough transform, Image Thresholding, or Texture analysis can be used as a method of recognizing a boundary of a plate or table.

The step S130 is a step of dividing only a region where food exists in the preprocessed learning food photograph, and can be manually divided by the learner. That is, it is possible to extract the food area more accurately by visually identifying and dividing the area where the food exists in the learning food photograph. In addition, for the learning of the amount of food, the divided food area can be subdivided into a circular area of interest of a predetermined size, for example, 10 10 pixels.

In operation S140, image features are extracted from the segmented food area and the ROI, and image features for color, shape, and texture are extracted.

Table 1 below is a table showing image characteristics extracted in step S140, and image characteristics of colors, shapes, and textures extracted by referring to the table will be described.

Image feature Detailed image features color Color (per R, G, B) Mean SD The histograms (per R, G, B) Mean SD Skewness Kurtosis shape Run-length SPE LPE Texture Tilt histogram Mean SD GLCM ASM Contrast Correlation IDM Entropy Inertia

Here, SD is the Standard Deviation, SPE is the Short Primitive Emphasis, LPE is the Long Primitive Emphasis, GLCM is the Gray Level Co-occurrence Matrix, ASM is the Angular Second Moment, and IDM is the Inverse Differential Moment.

The image characteristic for the color is an RGB color average value and a deviation and an RGB color histogram for the food region and each ROI, and the histogram has mean, standard deviation (SD), asymmetry Skewness, and Kurtoses histogram values.

The image characteristic of the shape is a number of pixels and a run length. The run length is composed of SPE (Short Primitive Emphasis) and LPE (Long Primitive Emphasis) It is used to measure.

The image characteristic of the texture is a gray level co-occurrence matrix (GLCM) texture parameter, which is one of a slope histogram and a texture analysis technique of the average and standard deviation. In other words, six texture variables and slope histograms of GLCM's Angular Second Moment (ASM), Contrast, Correlation, Entropy, IDM (Inverse Difference Moment) and Inertia are obtained and obtained as image features for texture.

In step S150, the automatic classifier learns the type and amount of food having the image characteristics acquired in step S140. That is, the type and amount of each food of the learning food photograph already known to the learner are input to the automatic classifier to generate the automatic classifier.

The automatic classifier may be a Bayesian classifier, a SVM (Support Vector Machine), or an ANN (Artificial Neural Network). Each automatic classifier creates an automatic classification structure by using the characteristic values of the images automatically extracted from the images by using the inherent principle. It is the end result of learning to create an automatic classifier that can classify the type and quantity of these foods.

In the case of automatic classifier learning, it is possible to estimate more accurately about kinds and amount of food when generating automatic classifier using many learning pictures about various kinds and amounts of food.

If the automatic classifier is generated through the above steps S110, S120, S130, S140, and S150, the kind and amount of the food are estimated using the inputted food photograph, Calculate the calories of each food. The food photograph input here means a food photograph that a user inputs to calculate calories.

Hereinafter, a method of estimating the type and amount of food of the input food image and calculating the calories based on the estimation will be described in detail.

First, the method of extracting the image feature from the input food photograph is almost the same as the method of extracting the image feature of the automatic classifier generating method described above. This is because the same image characteristics used in the generation of the automatic classifier must be obtained from the input food photograph to estimate the type and amount of food. Therefore, the image segmentation step S220, the food type and amount estimation step S240, and the calorie separation step S240, which are different from each other except the preprocessing step S210 and the image feature extraction step S230, Only the calculation step S300 will be described.

In step S220, a food region and a region of interest are segmented in the preprocessed input food photograph, and the automatic segmentation is used instead of the manual segmentation used in the automatic classifier generation. In the present invention, only the part where food exists is divided in the food photograph inputted by using Template Matching and Circle Hough Transform. That is, the food portion of the bowl or the plate is recognized in the food photograph inputted using the above method.

The step S240 is a step of inputting the extracted image features to the automatic classifier and estimating the type and amount of food present in the food photographs to be inputted. The automatic classifier compares the inputted image characteristics with the image characteristics of the learned food type and amount, and estimates the type and amount of food present in the inputted food photograph.

More specifically, the type of food is estimated using the image characteristics of the entire divided food region, and then the amount of food is estimated using the image characteristics of each region of interest in the estimated food type. Here, since the region of interest is a region in which the food region is re-divided into a predetermined size, more detailed estimation of the amount of food becomes possible.

The method of estimating the amount of food comprises the steps of first dividing a food region estimated as a kind of food into a plurality of regions of interest having a predetermined size, Estimate the amount of food that is actually present and estimate the size of the region of interest. That is, in the case of the food area, since the boundary of the food on the table is detected, the size of the food area may be different from that of the actual food. Similarly, by analyzing the image characteristics of food photographs, it is possible to estimate the food quantity more accurately than when extracting only the area where the food actually exists and estimating the amount of food based on the extracted area.

In addition, the type of food can be estimated using the CBIR (Content Based Image Retrieval) method as well as the food type estimation using the automatic classifier to more accurately estimate the type of food.

The step S300 is a step of calculating the calorie of the food using the automatic classifier when the type and amount of the food are estimated, and the calorie of the food is calculated using the database in which the calories per kind of the food are stored do.

In addition, if a food photograph before and after the intake is inputted, the type and amount of food of each food photograph may be estimated in step S240, and the calories consumed may be calculated using the difference in food amount of the food photographs in step S300.

In addition, the present invention can visually display the calories of each food calculated in step S300 on the corresponding food of the inputted food photograph. The display may display not only the calorie of each food but also the total calories added to the calories of the food, and the calories ingested may also be displayed. Therefore, the user can intuitively confirm not only the total calories consumed when the food is consumed, but also the calories actually consumed.

FIG. 11 is a block diagram showing a food habit management system using a calorie automatic calculation method using food photographs according to the present invention, and FIG. 12 is a block diagram illustrating a configuration of a food estimation module according to the present invention.

11 and 12, the eating habit management system 1000 using the calorie automatic calculation method using food photos according to the present invention includes a user terminal 100, a database 200, a food estimation module 310, A calculation module 320 and a dietary management module 330. [

The user terminal 100 may be a smart phone, a desktop or a notebook for uploading food photos on a web or an app. The user terminal 220 can communicate with the server 230 via the network 10 in a wired or wireless manner. A user uploads a food photograph on the web or downloads an app, installs a food image on the application can do. In addition, the user can additionally input his or her age, weight, and key.

The database 200 stores calorie data for each set type of food, and is used to calculate the calories of the food using the type and amount of the food estimated by the food estimating unit 313.

The food estimation module 310 of the server 300 includes an automatic classifier unit 311 generated through machine learning on the type and amount of food, an image feature extraction unit 312 that extracts image features from a food photograph, And a food estimating unit 313 for estimating the kind and quantity of food using the automatic classifier unit.

 In addition, the image feature extraction unit 312 includes a preprocessing unit 312a for preprocessing input food photographs, an image division unit 312b for dividing a food region and a region of interest in the preprocessed input food image, And a feature extraction unit 312c that extracts image features of the ROI.

The automatic classifier unit 311 is a place where an automatic classifier for each type and amount of food is generated by using a picture of learning food, and the learner prepares the classifier by learning in advance using the learning food picture. The image features used for the learning are the same as the image features used in the step S140 described above, so a detailed description thereof will be omitted.

The image feature extraction unit 312 extracts image features necessary for estimating the type and amount of food in the inputted food photograph.

The preprocessing unit 312a of the image feature extraction unit 312 performs a preprocessing process to facilitate the extraction of image features of the input food photographs. The preprocessing process resizes , Rotating and dividing the plate or table portion in the incoming food photograph. In addition, a line Hough transform, image thresholding or texture analysis technique may be used to segment the plate or table portion.

The image dividing unit 312b divides the food region, which is the region where the food exists, in the preprocessed food photograph, and re-divides the divided food region into the region of interest having the predetermined size. The food area is, for example, a portion in which food is contained in a plate if the input food photograph is a photograph of the plate, and a bowl in which the food is placed if the food photograph is a food placed on the table. The region of interest refers to a region of a predetermined size predetermined for estimating the amount of food.

The feature extraction unit 312c extracts image features for color, shape, and texture for each of the segmented food area and the ROI.

The food estimating unit 313 estimates the type and amount of the food present in the food image inputted using the image characteristic extracted by the image characteristic extracting unit 312 and the automatic classifier generated by the automatic classifier unit 311 . That is, the food estimating unit 313 first estimates the kind of food of each food region using the image characteristic of each food region, and estimates the type of the food using the image characteristic of each region of interest of the estimated food region Estimate the amount.

In addition, the food estimation module 310 can estimate the type of food by further using a content-based image search technique as well as an automatic classifier to more accurately estimate the type of food.

The calorie calculation module 320 calculates the calories of the foods in the uploaded food photographs using the type and amount of food estimated in the food estimation unit 313 and the calorie data of the database 200. [

In addition, the eating habit management system 1000 using the calorie automatic calculation method using the food photograph according to the present invention may be configured such that the calories calculated by the calorie calculation module 320 are stored on each food of the food photographs uploaded from the user terminal 100 Can be displayed.

The eating and drinking management module 330 guides and manages the user's eating habits by using the type of food consumed by the user, the amount consumed, and the calories consumed.

The eating and drinking management module 330 may be modified by the administrator so that various functions are provided according to the purpose of use.

For example, we can estimate the type and amount of food in user-uploaded food photographs, and then use the estimated food type and amount to determine the type of food that the user prefers and then use it to analyze the user's nutrient intake status Which can be used to guide poor nutrients to the user or to guide the user through the addition of nutrients for a balanced nutrient intake.

Alternatively, the calorie consumed by the user on a daily, weekly, and monthly basis can be analyzed to provide a food intake plan for weight control. In order to provide the above function, the eating habits management module 330 may further use information about the age, weight, and key input by the user.

Alternatively, the eating and drinking management module 330 may be used for personalized consulting for dieting or for consulting the eating habits that are appropriate for the patient's eating habits and disease characteristics in a hospital.

According to the eating and drinking management system 1000 using the method according to the present invention, the user can check the calories of the food he / she has ingested in real time. Based on the type, quantity and calorie of the food he / she has ingested, Is provided.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. Modification, substitution, addition, or the like.

100: user terminal 200: database
300: server 310: food estimation module
311: Automatic classifier unit 312: Image feature extraction unit
312a: preprocessing unit 312b:
312c: feature extraction unit 313:
320: Calorie Calculation Module 330: Diet Management Module
1000: Eating management system

Claims (12)

A user terminal for uploading food photographs before and after food intake on a web or an app (app), a database storing calorie data for each kind of food, a server including a food estimation module, a calorie calculation module and a dietary management module Wherein the server of the eating and drinking management system comprises:
(a) calculating an image feature for color, texture and shape of a study food photograph;
(b) generating an automatic classifier in which the type and amount of food for the learning food photograph are learned through machine learning based on the calculated image characteristic values;
(c) extracting image characteristics of colors, textures, and shapes from the input food photographs, and inputting the image features extracted from the input food photographs into the automatic classifiers to estimate the types and amounts of foods; And
(d) calculating calories by using the type and amount of the estimated food; , ≪ / RTI &
The step (c)
(c1) changing an inputted food photograph to a set resolution, and dividing a table or a plate in the food photograph;
(c2) dividing the food region in which the food exists in the divided food photographs and re-dividing each divided food region into a plurality of regions of interest having a predetermined size;
(c3) extracting image features of color, shape, and texture by image processing the divided food area and the ROI;
(c4) inputting the image characteristic of the entire divided food region to the automatic classifier to estimate the type of food; And
(c5) inputting the image characteristic of the region of interest with respect to the food region in which the food type is estimated into the automatic classifier, and estimating the amount of the estimated food type,
Wherein the color image feature is an RGB color histogram, the shape image feature is a number of pixels and a run length, and the texture image feature is a texture parameter of a gradient histogram and a GLCM (gray-level co-occurrence matrix) Automatic Calorie Calculation Method Using Food Photographs Characterized. delete The method according to claim 1,
Wherein the server of the eating habit management system divides a table or a plate using at least one of a linear Huff transformation, an image thresholding, and a texture analysis in the step (c1). The method according to claim 1,
Wherein the server of the eating habit management system divides the food region using template matching and circular Huff transform in the step (c2). The method according to any one of claims 1 and 3 to 4,
Wherein the server of the eating habit management system in the step (c) estimates the type of food by further using a content-based image search. 6. The method of claim 5,
Wherein the server of the eating and drinking management system calculates the calories of the food consumed through photographs before and after the food intake in the step (d). The method according to claim 6,
Wherein the server of the eating and drinking management system displays the calculated calories visually on each food of the inputted food photograph in the step (d). A user terminal for uploading food pictures before and after food intake on a web or an app (app);
A database storing calorie data for each type of food; And
A food estimating module for extracting image characteristics from food photographs before and after the food is uploaded from the user terminal and inputting the image characteristics into an automatic classifier previously learned about the type and amount of food to estimate the type and quantity of food of the food photographs ; A calorie calculation module for calculating an intake calorie using a difference between an estimated food type and a pre-intake and post-intake food amount and calorie data of a database, and a dietary habit management module for guiding and managing a dietary habit of the user using the calculated intake calorie ; ≪ / RTI >
The food estimation module includes an automatic classifier for mechanically learning the type and amount of food based on image characteristics of colors, textures, and shapes extracted from a photograph of a food for learning, and an image classifier for classifying colors, textures, And a food estimating unit for inputting the image characteristics of the extracted color, texture, and shape into the automatic classifier unit and estimating the type and amount of the food,
The image feature extracting unit may include a preprocessing unit for changing input food photographs to a set resolution and dividing a plate or a table in a food photograph, a food region in which food is present in the food photographs divided by the preprocessing unit, And a feature extracting unit for extracting image features of color, texture, and shape in the food area and the ROI,
The food estimating unit inputs the image characteristic of the entire food region into the automatic classifier to estimate the type of the food, and then the image characteristic of the region of interest with respect to the estimated food type is classified into the automatic classifier And estimates the amount of the estimated food type,
Wherein the color image characteristic is an RGB color histogram, the shape image characteristic is a pixel number and a run length, and the texture image characteristic is a texture parameter of a gradient histogram and a GLCM (gray-level co-occurrence matrix) Food management system using calorie automatic calculation method using food photographs. 9. The method of claim 8,
Wherein the preprocessing unit divides a food plate into at least one of a food image to be inputted and a food image using at least one of line-Hough transform, image thresholding, and texture analysis. Eating management system using method. 9. The method of claim 8,
Wherein the image dividing unit divides the food area using template matching and circular Huff transform. 11. The method according to any one of claims 8 to 10,
Wherein the food estimation module further estimates the type of food by using a content-based image search. 12. The method of claim 11,
Wherein the calorie calculation module displays the calories of the food on each food of the food photo inputted from the user terminal.

Images