KR20180093141A - A meal calendar system using the image processing method based on colors - Google Patents

Abstract

The present invention relates to a meal diary system by analysis of a food image based on color, which records a meal diary by photographing a user′s menu by using a smart terminal and analyzes the menu on the basis of the photographed menu image based on color. Also, the meal diary system by analysis of a food image based on color recommends the menu corresponding to the user in accordance with an analysis result. The meal diary system by analysis of a food image based on color comprises: a meal image receiving unit receiving the menu image photographed by a camera; a menu color sorting unit sorting the menu color in the menu image by cluster analysis; a depth learning forming unit learning and forming a depth nerve network sorting food by a menu color feature by a deep learning technique; a menu food sorting unit sorting the food based on the menu color by the depth nerve network; and a menu food analysis unit extracting nutrient evaluation and recommended food on the menu. According to the present invention, the meal diary system can deduct an effective result by applying a big data analysis technique such as the deep learning technique or cluster analysis by clearly setting an analysis element on analysis of a menu image based on five base colors.

Description

[0001] The present invention relates to a meal-calendar system,

The present invention relates to a color-based food image analysis, in which a user of a smart terminal photographs his / her diet and records a meal diary, analyzes the diet based on the color of the taken food image, and recommends a user- The present invention relates to a meal diary system.

In general, the eating habits for proper nutrient intake and calorie control are the most basic ones to protect physical health, but due to the increase in the nuclear family and the social activities of women, eating habits are changed and eating habits become unfavorable. Obesity, diabetes, hypertension And various lifestyle-related diseases such as a disease are becoming a problem.

Therefore, it is very important to investigate the eating habits and activity habits of the individual in relation to the nutritional status analysis of the subject, the correction of lifestyle, and the improvement of diseases such as obesity, diabetes and hypertension. Through the analysis of eating habits, you can see the foods you usually enjoy, the nutrients you eat a lot, the nutrients you lack, and the total amount of calories consumed. In addition, it is possible to judge whether the balance between the calories consumed through meals and the calories consumed through activities. Physicians, nutritionists, and exercise prescribers can use the results of these analyzes to improve their eating habits and to prescribe meals and exercise prescriptions that can supplement deficient activities or exercises.

In the investigation of these eating habits, the existing methods have been mostly utilized by one-on-one counseling or using questionnaires. One-on-one counseling was a form in which the patient's diet and activity habits were interviewed by a nurse or doctor directly in front of the patient, but it was difficult to be meticulously and precisely realistic in terms of time constraints. Due to this difficulty, a method of providing a predetermined form of diary questionnaire to the patient in advance and filling it out in advance was used. The meal diary form is a table in which the date, the day of the week, and the meal are separated, and the subject directly describes the meal contents corresponding to the date or day of the week and the meal. [Patent Document 1]. However, the method using the questionnaire is advantageous in that it can be practically and precisely investigated by the freedom of description. On the other hand, the kind of food or the kind of activity is expressed without restriction, so that subjective judgment of the amount of intake, Problems may arise due to unclear transmission or the like. In particular, it is time consuming and tedious to calculate the nutrient intake rate, calorie, etc. by calculating the amount of food consumed and the activity time of each person based on many researches.

Therefore, in order to solve such a problem, a technique of providing diary diary by using an application has been proposed [Patent Document 2, 3, 4]. That is, in the related art, the user selects a date on a calendar by using a mobile communication terminal such as a smart phone, which is always carried by the user, inputs a diet, records the dietary habits, and analyzes and displays the recorded content. In addition, a technology for automatically estimating the calorie for a meal is provided when the meal information is recorded. Among these techniques, the camera is used to photograph the food, but the image of the food is manually analyzed by a specialist such as a video engineer. In addition, a technology for evaluating the diet by analyzing the image data of the diet has been proposed, but a specific means for evaluating the diet has not been proposed [Patent Document 5].

[Patent Document 1] Korean Published Patent Application No. 10-2005-0029903 (published on March 29, 2005) [Patent Document 2] Korean Patent Laid-Open Publication No. 10-2015-0034426 (published on April 03, 2013) [Patent Document 3] Korean Published Patent Application No. 10-2015-0016046 (published Feb. 11, 2015) [Patent Document 4] Korean Published Patent Application No. 10-2007-0107970 (published on November 11, 2007) [Patent Document 5] Korean Published Patent Application No. 10-2009-0046991 (published on May 12, 2009)

It is an object of the present invention to solve the above-mentioned problems, and it is an object of the present invention to provide a smart terminal which records a meal diary of a self terminal, records a meal diary, analyzes the dietary frame based on color, Based diary image analysis, which recommends a proper diet for the diary.

It is another object of the present invention to provide a method and apparatus for analyzing a main color by performing a cluster analysis based on a basic color (red, yellow, green, black, white) based food image analysis, which is analyzed by a color-based learning method.

In order to accomplish the above object, the present invention is directed to a meal diary system through color-based food image analysis, comprising: a meal image receiving unit for receiving a meal image taken by a camera; A menu color classification unit for classifying the color of the meal into cluster analysis in the menu image; An in-depth learning forming unit for learning a deep neural network that classifies food as a meal color feature by a deep learning method; A dietary food classifying unit for classifying the food from the color of the diet by the neural network, and a dietary food analyzing unit for nutritionally evaluating the dietary food and extracting the recommended food.

Further, the present invention is characterized in that, in a meal diary system through color-based food image analysis, the color of the meal is divided into five basic colors of red, yellow, green, black and white.

Further, in the present invention, in a meal diary system through color-based food image analysis, the cluster analysis is performed first by color segmentation, and a decision tree is constructed through un-supervised learning And a cluster analysis is performed.

In addition, the present invention is characterized in that, in a meal diary system through color-based food image analysis, the community analysis is performed to maximize the degree of cohesion in the community and to maximize the degree of separation between the communities.

Further, in the meal diary system according to the color-based food image analysis, the deep learning forming unit learns the food color of the food, and outputs the food or the food type as a neural network structure, And a plurality of layers are divided into layers.

As described above, according to the diary diary system through the color-based food image analysis according to the present invention, when the diary is simply photographed by the smart terminal, the diary is automatically analyzed and the contents of the diary are not described in detail. The effect of analyzing the diary can be obtained easily.

In addition, according to the meal diary system through the color-based food image analysis according to the present invention, the analysis of the food image can be performed by clearly setting the analysis elements based on the five basic colors, It is possible to obtain an effective result by applying a big data analysis technique.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram of an entire system for implementing the present invention. FIG.
FIG. 2 is a block diagram of a configuration of a meal diary system through color-based food image analysis according to an embodiment of the present invention; FIG.
Fig. 3 is a diagram showing a principle of a cluster analysis used in the present invention. Fig.
Figure 4 is a visualization of the cohesion and the degree of separation of the graph-based clustering according to the present invention.
Figure 5 is a visualization of the degree of cohesion and separation of the prototype-based clustering according to the present invention.
6 is a diagram showing a classification of a cluster analysis standard according to the present invention.
7 is a structural diagram of a depth-of-field network according to an embodiment of the present invention.
8 is a diagram illustrating a process of analyzing a neural network according to an embodiment of the present invention.
9 is a block diagram of a configuration of a diary client according to an embodiment of the present invention;
10 is an exemplary view of a meal schedule screen according to an embodiment of the present invention;
FIG. 11 is an exemplary view of a meal-taking screen according to an embodiment of the present invention; FIG.
FIG. 12 is an exemplary diagram of a food screen and a result screen according to an embodiment of the present invention; FIG.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the drawings.

In the description of the present invention, the same parts are denoted by the same reference numerals, and repetitive description thereof will be omitted.

First, an example of the overall system configuration for implementing the present invention will be described with reference to FIG.

1, the overall system for implementing the present invention includes a smart mobile terminal 10 such as a smart phone of a user, a diary diary client 20 installed in the smart mobile terminal 10, And an analysis server 30 for analysis. The smart mobile terminal 10 and the analysis server 30 are connected to each other through a network 80 such as the Internet. In addition, a database 40 for storing data in the analysis server 30 may be additionally configured.

The smart mobile terminal 10 is a mobile terminal having ordinary computing functions such as a smart phone and a tablet PC used by a user, and has a camera capable of capturing a menu image. The diary diary client 20 is a mobile application (or an application or an app) installed and executed in the smart mobile terminal 10.

The user executes the client 20 installed in the smart mobile terminal 10, captures and records the diet of the meal he / she has eaten with the camera. At this time, preferably, the user simply records the information on the meal according to the instructions of the client 20. [ In addition, the client 20 accesses the server 30 to transmit the recorded image of the menu, or transmits the meal information for the menu image, for example, the meal time, the location, and the meal information directly input by the user. In addition, the client 20 can receive the analysis result of the meal diary from the analysis server 30 and display the result on the display of the smart mobile terminal 10.

Hereinafter, since the work performed by the diary client 20 or the user is performed by the smart mobile terminal 10, it is described that the client 20 or the smart mobile terminal 10 performs the operation for convenience of explanation. do. That is, the client 20 or the smart mobile terminal 10 is mixed. In addition, input data for a user to process a diary entry is inputted through an input device of the smart mobile terminal 10, and the processing result is outputted through an output device such as a screen of the smart mobile terminal 10. Hereinafter, the description of a user performing a task is an operation performed through the smart mobile terminal 10.

The analysis server 30 is a normal application server and is a server that manages or analyzes diary diaries. In particular, the analysis server 30 receives the meal information such as a meal image taken or recorded by the user from the client 20, and analyzes and evaluates each meal of the meal information. In addition, the analysis server 30 instructs the client 20 to analyze the results of the dietary analysis and recommend dietary contents.

Specifically, the analysis server 30 analyzes each meal image by color-based clustering and automatically evaluates each meal or meal according to the analysis result. That is, in one embodiment, the analysis server 30 automatically evaluates the data of each meal according to a predetermined rule.

In addition, the analysis server 30 may calculate the recommended diet content for the user in the future using the evaluation of the meal diary of the user, and transmit the recommended content of the calculated diet to the client 20 or the smart mobile terminal 10 send.

Meanwhile, the diary client 20 and the analysis server 30 may be implemented according to a method of configuring a normal client and a server. That is, the functions of the entire system can be shared according to the performance of the client, the amount of communication with the server, and the like. For example, the client 20 may simply photograph the image of the menu and the server 30 may perform all operations such as preprocessing and analysis of the menu image. Alternatively, the client 20 may perform all tasks, such as image analysis of the food image as well as the image capture of the diet, and the server 30 may perform only data or backup functions. Alternatively, the client 20 may take a diet image and perform a simple analysis, but the server 30 may share a large data analysis or a deep run. Hereinafter, the meal diary system will be described, but it can be implemented in various forms according to the server-client configuration method.

The database 40 includes a user DB 41 for storing registered user information, a meal record DB 42 for storing a meal diary such as a meal image recorded by the user, a meal list to be evaluated and recommended for each diary of each user And a DB 43 for storing analysis results. However, the configuration of the database 40 is only a preferred embodiment. In the development of a specific device, the database 40 may have a different structure in consideration of ease of access and retrieval, efficiency, and the like.

Next, the color of the menu used in the present invention will be described in more detail.

The basic color (or basic food color) used in the present invention is divided into five colors such as red, yellow, green, black, and white. Each base color of the diet color has a deep correlation with the nutrition of each food. Therefore, it is possible to analyze the nutritional aspects of the diet with only the color of the diet. An example of the relationship between nutrition for each diet color is as follows.

First, red color has many elements that encourage the energy of the heart. Lycopene has excellent anticancer activity, and it helps immunity and vascular health. Anthocyanin also has 10 stronger anti-inflammatory actions than aspirin and is effective against cancer and arteriosclerosis.

Next, yellow helps stomach health and digestion. Carotinoids have anticancer, antioxidant and anti-aging properties. It also helps prevent adult diseases and improve blood circulation. Yellow is a symbol of energy that activates motor nerves and is known to be effective in creating energy for muscles and is often used in arthritis treatments.

Next, the green color relieves liver fatigue and is effective in lung waste. It also has the effect of stabilizing and alleviating the nervous system. Chlorophyll and vitamin C have a detoxifying effect by helping metabolism, helping regenerate cells, and absorbing harmful substances such as heavy metals in the body and discharging them out of the body.

Next, black smoothes the kidney, bladder and genital system functions. Anthocyanin is an abundance of cell-protecting functions that prevent immunity and aging, give skin elasticity, and reduce eye fatigue to protect your eyesight.

Next, white prevents exposure to lung and respiratory diseases. Anthoxanthine (Flavonoid), a pigment that gives white color, suppresses Chernic Oxidation and releases harmful substances out of the body, enhances immunity and increases resistance to various bacteria and viruses.

Next, a meal diary system through color-based food image analysis according to an embodiment of the present invention will be described with reference to FIG. The meal information recording system according to the present invention is described as a system implemented in the analysis server 30 in advance, but all or some of these functions may be implemented in the client 20.

2, the meal information recording system 30 according to the present invention includes a meal image receiving unit 31 for receiving a food image taken by a camera, a food color classification unit 32 for sorting food color in the food image, , An in-depth learning forming unit (33) for forming a depth neural network for classifying food into color features by a deep learning method, a food learning classifying unit (33) for classifying food from the color of the food by the neural network, (34), and a diet food analysis unit (35) for extracting nutritional evaluation and recommended foods with respect to the diet.

First, the meal image receiving unit 31 receives the image of the meal taken by the camera. The diet image is an image captured by the camera of the smart mobile terminal 10 in advance.

Next, the menu color classification section 32 classifies the menu color in the menu image. At this time, the color of the diet is classified by five basic colors such as red, yellow, green, black and white. It is also preferably classified by a cluster analysis method.

Cluster analysis is a method of grouping clusters of similar or homogeneous data without knowing the cluster to which the data belongs. Cluster analysis is used extensively in the initial exploratory analysis phase for the purpose of identifying and utilizing meaningful or useful data structures or understanding the structure of data.

Color segmentation is performed first by cluster analysis, and a decision tree is constructed through un-supervised learning to perform cluster analysis. Preferably, a semi-supervised learning approach can be applied to classify communities made through cluster analysis.

On the other hand, if the data consists only of two dimensions of X and Y, it is possible to grasp the pattern, the presence of cluster, and the relationship between the data in the dataset just by visual inspection by drawing a scatter plot. 3D can also be visually confirmed by drawing a 3D plot. However, since it becomes difficult to judge whether or not there is a community by human eyes, it is necessary to use cluster analysis at this time.

In the cluster analysis, the criteria of model generation and evaluation mainly use the cohesion and separation scales. As shown in FIG. 3, the clustering principle is expressed with a formula and a scale so that the computer understands the cluster analysis. The cluster is formed to maximize the degree of cohesion in the cluster and maximize the separation between the clusters.

As shown in Fig. 3, the sum of the distance between the data of the cluster 1 and the centroid c ₁ is minimized, and the sum of the distance between the data of the cluster 2 and the centroid c ₂ is minimized. Further, as shown in ②, thereby maximizing the center (centroid) of the cluster center c ₁ and 2 (centroid) and the distance c ₂ of the first cluster.

In other words, the basis of the cluster analysis algorithm includes the concept of optimization.

Graph-based clustering and Prototype-based clustering are described in detail.

First, the concept of cohesion and separation of graph-based clustering is expressed as follows.

[Equation 1]

This is shown in FIG. 4 as a visualization.

Also, the concepts of cohesion and separation of prototype-based clustering are expressed as follows. Prototype means the centroid or medoid of a cluster.

&Quot; (2) "

Here, the degree of cohesion is obtained by sum of squares (SSE).

This is shown in Fig. 5 as a visualization.

On the other hand, there are several criteria that can distinguish cluster analysis.

As shown in FIG. 6, it can be divided into hierarchical clustering and partitional clustering depending on whether there is a partial cluster in a cluster or not. Hierarchical clusters have partial clusters in one cluster, whereas partitional clusters are exclusive of each other without overlapping or overlapping clusters.

Hierarchical cluster analysis is based on the aggregation method (Aggolomerative, Bottom-up method), which starts from individual data and ties together similar data to a cluster, and that all data belongs to one cluster, (Divisive, top-down method).

Partitional clustering methods include distance-based clustering, density-based clustering, and neural network-based clustering.

The food color classification unit 32 classifies the color classification unit into a synthetic color classification system.

That is, the image or picture is divided into pixels, converted into RGB color, and classified into five colors based on the cluster analysis algorithm.

The RGB color values of each basic color are as follows.

Red color: (R: 255, G: 000, B: 000)

Yellow: (R: 255, G: 255, B: 000)

Green: (R: 000, G: 255, B: 000)

Black: (R: 000, G: 000, B: 000)

White: (R: 255, G: 255, B: 255)

Next, the in-depth learning forming unit 33 forms a depth neural network that classifies food into a food color feature by a deep learning method.

The Deep Learning method is defined as a set of machine learning methods that try to achieve a high level of abstractions through a combination of various nonlinear transformation techniques. It is a field of machine learning.

Deep Neural Network (DNN) is formed by learning by in-depth learning method. The structure of the deep layer neural network is shown in FIG. The in-depth neural network as shown in Fig. 7 is formed by learning, and the in-depth neural network can be further evolved so that sample data or actual data can be accumulated.

As shown in FIG. 8, the in-depth learning forming unit 33 learns the food color by inputting the color of the food, and outputs the food or the food type as a neural network structure. In particular, preferably, the deep learning forming unit 33 is divided into a plurality of neural network layers to efficiently learn a plurality of layers.

Traditional machine learning has the advantage that people can easily understand features. Since a person directly designates a feature value, it is relatively easy to analyze the machine learning algorithm because it can easily understand how to process the input and output the result through a calculation process. However, in cases where unlabeled data is to be solved by unsupervised learning, it is often not known how to approach meaningful feature values. In this case, the benefits of in-depth learning are advantageous. It is a practical limitation to directly design feature values when the amount and type of data are increasing.

Next, the dietary food classification unit 34 classifies the food from the color of the meal using the above-described in-depth neural network. That is, the diet food classifier 34 inputs the classified color of the diet into the neural network, and classifies or extracts food by its output.

Next, the diet food analyzer 35 performs the nutritional evaluation according to each meal according to the classified food and the color of the classified diet. And nutritional assessment and food recommendations based on the color evaluation of the diet.

Next, a method of interfacing a meal diary system on a smart terminal according to an embodiment of the present invention will be described with reference to FIGS. 9 to 12. FIG.

9, the diary diary client 20 includes an interface unit 28 for receiving a command of a user or outputting a menu or the like, a meal schedule management unit 21 for managing a meal schedule to be recorded, And a meal recording transfer section 23 for transferring the meal record to the server 30, such as a stored meal image. In addition, it may further comprise a record display unit 24 for summarizing or detailing the meal record and analysis contents on the screen, or an alarm unit 24 for notifying the meal recording schedule with an alarm. Further, it further includes a storage unit 29 for storing data. The storage unit 29 provides a storage space using a storage device such as a memory of the user terminal 10. [

First, the interface unit 28 provides a screen (or a user interface) for receiving a user command. The interface unit 28 displays a calendar screen, a meal diary shooting screen, a meal diary result screen, and the like on the screen, and receives the taken food image. In particular, the interface unit 28 provides a normal touch screen user interface (UI) or a touch graphic user interface (GUI). Hereinafter, a part of the functions of the meal recording client 20 that need to be interfaced with the user receives the input through the interface unit 28 or outputs the result. 10 is an example of a screen displaying the schedule of the meal diary to photograph the meal.

Next, the meal schedule management unit 21 manages the meal schedule to be recorded. The meal record schedule is a schedule for meals to record meal information. The meal consists of at least three meals such as breakfast, lunch, and dinner, and can include additional meals such as snacks. The meal record schedule is set for each meal of the day on a daily, monthly or weekly basis.

Next, the meal photographing section 22 photographs the meal that has been eaten by the person in the meal, and records the contents of the meal. And receives the image of the meal taken by the camera. In addition, the meal recording unit 22 can additionally input the food name, meal amount, memo, etc. in a predetermined format. 11 shows an example of a photographed menu image.

Next, the meal record transfer section 23 transfers the meal record, such as a meal image recorded and stored by the meal record recording section 22, to the server 30. At this time, the meal record is transmitted to the server 30 periodically or by a command of the user.

Further, the result display section 24 displays the recorded and stored meal information or analysis result on the screen. At this time, the user can intuitively view the food using a table or a graph. For this purpose, the meal information can be summarized or processed and displayed by statistical processing. 12 is an example of a screen displaying a food image and an analysis result.

Next, the alarm unit 25 outputs an alarm to make a meal diary when the time of the corresponding meal arrives, or when the meal-taking time elapses.

Although the present invention has been described in detail with reference to the above embodiments, it is needless to say that the present invention is not limited to the above-described embodiments, and various modifications may be made without departing from the spirit of the present invention.

10: smart mobile terminal 20: client
21: Meal schedule management unit 22:
23: Meal record transfer unit 24: Result display unit
25: alarm unit 28: interface unit
30: Meal diary system
31: a food image receiving unit 32: a food color classification unit
33: deep learning forming unit 34:
35: Diet Food Analysis Department
40: Database 80: Network

Claims (5)

In a meal diary system through color based food image analysis,
A meal image receiving unit for receiving a meal image taken by a camera;
A menu color classification unit for classifying the color of the meal into cluster analysis in the menu image;
An in-depth learning forming unit for learning a deep neural network that classifies food as a meal color feature by a deep learning method;
A diet food classifying unit for classifying food from the color of the meal by the depth neural network,
And a dietary food analysis unit for extracting nutritional evaluation and recommended foods with respect to the diet.
The method according to claim 1,
Wherein the color of the meal is divided into five basic colors of red, yellow, green, black, and white.
The method according to claim 1,
Wherein the cluster analysis is performed by first performing color segmentation, and building a decision tree through un-supervised learning to perform cluster analysis. system.
The method of claim 3,
Wherein said cluster analysis maximizes cohesion within a cluster and forms a cluster to maximize the degree of separation between the clusters.
The method according to claim 1,
Wherein the depth learning forming unit learns the color of the food based on the color of the food, and learns the food or the food type as the output, Diary system through.

Images