KR20210099876A - prsonalized nutrition and disease management system and method using deep learning food image recognition - Google Patents

Abstract

The present invention relates to a personalized nutrition and disease management system using deep learning-based food image recognition and a method thereof and, more specifically, to a personalized nutrition and disease management system using deep learning-based food image recognition, which acquires a food image by using a camera in a smart device, determines the acquired food on the basis of a deep learning algorithm, generating management information according to the food, and providing personalized nutrition and disease management solutions based on the management information, and a method thereof. According to the present invention, the system provides an effect of providing nutrition management and disease management services based on a food additive database and a nutrition database as a service through a smartphone application by using an object recognition algorithm for an image captured by a camera. The system comprises a user terminal and a nutrition management server.

Description

Personalized nutrition and disease management system and method using deep learning algorithm-based food image recognition {prsonalized nutrition and disease management system and method using deep learning food image recognition}

The present invention relates to a system and method for personalized nutrition and disease management through food image recognition based on a deep learning algorithm, and more particularly, a deep learning algorithm obtained by acquiring a food (food) image using a camera in a smart device A personalized nutrition and disease management system through deep learning algorithm-based food image recognition that can provide personalized nutrition and disease management solutions based on the generation of food-dependent management information after determining the food (food); it's about how

As life expectancy increases, interest in healthy food intake is also increasing.

As people's quality of life improves, average height increases and life expectancy increases, but the reality is that there is a lack of knowledge about food (food) related to individualized nutritional information and disease prevention, resulting in incorrect health management. am.

Excessive nutritional intake is also occurring, and accordingly, the incidence of diseases such as obesity, diabetes, high blood pressure, and hyperlipidemia is increasing.

Various platforms have been released to solve this problem, but nutritional intake information for health management is weak and consists of simple diet-oriented functions such as calorie management of food.

In addition, the usage rate is low due to the inconvenience of using the information on the food consumed in a text-oriented input method, and a user-centered simple input method platform construction is required.

Although it is necessary to provide customized information for each individual in nutrition management, current services are based only on fragmented information.

Along with the interest in food (food) consumption, interest in food additives, raw materials, and country of origin has rapidly increased. However, it is not easy to find detailed information on food additives and raw materials or evaluation of additive safety at present.

In addition, even if detailed information on food additives and raw materials is provided, it is not easy to determine whether food additives described in difficult terms are harmful.

KR10-2065657B1

The present invention was devised to solve such a problem, and when a user takes a picture of the food (food) eaten by the user and transmits it to the server, Nutri-Score nutritional grade and EWG food additive grade are provided based on the image information of the food (food) The purpose of this program is to provide a personalized nutrition and disease management system and method through deep learning algorithm-based food image recognition that can provide customized disease management information.

In addition, the present invention expresses the image of food (food) in pixels, and based on this, a deep learning technique is applied to provide the most similar food (food) information. Personalized through food image recognition based on a deep learning algorithm. It aims to provide a nutritional and disease management system and method.

In addition, the present invention provides an individual through food image recognition based on a deep learning algorithm that can provide information to select a desired food (food) in the order of high similarity when information on food (food) does not match within the image. It aims to provide customized nutrition and disease management systems and methods.

In addition, the present invention is based on a deep learning algorithm that recognizes images of various foods (foods) for each individual and provides nutritional grades and disease management methods through Nutri-Score nutritional grade and EWG food additive grade-based algorithm calculation. It aims to provide a personalized nutrition and disease management system and method through food image recognition of

In addition, the present invention provides an artificial intelligence-based image processing method that can recommend a realistic diet suitable for the user's taste, health condition, constitution, and religion while considering the balance of user's nutrient intake in conjunction with big data. The purpose of this study is to provide a personalized nutrition and disease management system and method through food image recognition based on a learning algorithm.

According to one aspect of a method for managing personalized nutrition and disease through food image recognition based on a deep learning algorithm according to the present invention for achieving the above object, at least one or more foods included in the food image information transmitted from the user terminal recognizing; analyzing food additives and nutritional information for the recognized at least one food; transmitting food additives and nutritional analysis results for the corresponding food to the user terminal; and transmitting, to the user terminal, customized diet information recommended based on pre-stored user nutritional information.

In addition, the step of recognizing at least one or more foods included in the food image information transmitted from the user terminal may include: extracting by designating a region of interest of at least one or more foods included in the food image information transmitted from the user terminal; and recognizing at least one food included in the food image information using a machine learning model learned based on the extracted region of interest.

On the other hand, according to one aspect of the personalized nutrition and disease management system through food image recognition based on a deep learning algorithm according to the present invention for achieving the above object, a user terminal for receiving nutrition and disease management services; and a nutrition management server that analyzes the food image information transmitted from the user terminal and transmits customized diet information to the user terminal based on the analyzed result and user nutrition information.

In addition, the nutrition management server is a network interface for communicating with the user terminal; an image recognition unit for designating and extracting a region of interest of at least one food included in the food image information transmitted from the user terminal; an image recognition model unit for recognizing at least one food included in food image information using a machine learning model learned based on the region of interest extracted from the image recognition unit; and a nutrition management unit that analyzes food additives and nutritional information for at least one food recognized by the image recognition model unit and recommends customized diet information based on pre-stored user nutritional information.

According to the present invention, there is an effect of providing a food (food) additive database, nutrition database-based nutrition management and disease management service by using an object recognition algorithm for a photographed image of a device camera as a service through a smartphone application. .

In addition, based on the image taken by the user, by using deep learning technology of artificial intelligence, it is grafted to the nutrition rating method and food additive rating calculation to provide the user with diet evaluation data in real time, and by artificial intelligence-based image processing method It has the effect of recommending a customized diet to the user.

1 is a view showing the overall configuration of a personalized nutrition and disease management system through food image recognition based on a deep learning algorithm according to an embodiment of the present invention.
Figure 2 is a diagram showing an example of image recognition and nutritional and ratio information analysis through the system of Figure 1;
3 is a diagram showing an example of object recognition by deep learning technology.
4 is a diagram illustrating a structure of a convolutional neural network (CNN).
Figure 5 is a view showing an operation flow for the image recognition unit and the image recognition model in the system of Figure 1;
6 and 7 are views showing an example of food image information photographed in a smart device.
8 is a view showing a personalized nutrition and disease management process through food image recognition between a user terminal and a nutrition management server.
9 is a view showing an example of Nutri-Score of products by brand.
10 is an example screen of the chemical additive grade display for each food.
11 is a view showing an example of the top 10 food additive labels that should not be eaten.
12 is a view showing an example of a nutritional component analysis for a specific food.
13 is a view showing another example of a method for personalized nutrition and disease management through food image recognition.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Prior to this, the terms or words used in the present specification and claims should not be construed as being limited to conventional or dictionary meanings, and the inventor should properly understand the concept of the term in order to best describe his invention. Based on the principle that it can be defined, it should be interpreted as meaning and concept consistent with the technical idea of the present invention. Therefore, the configuration shown in the embodiments and drawings described in the present specification is only the most preferred embodiment of the present invention and does not represent all the technical spirit of the present invention, so at the time of the present application, various It should be understood that there may be equivalents and variations.

1 is a view showing the overall configuration of a personalized nutrition and disease management system through food image recognition based on a deep learning algorithm according to an embodiment of the present invention, and FIG. 2 is image recognition and nutrition and It is a diagram showing an example of ratio information analysis, FIG. 3 is a diagram showing an example of object recognition by deep learning technology, FIG. 4 is a diagram showing the structure of a convolutional neural network (CNN), and FIG. 5 is FIG. It is a view showing the operation flow for the image recognition unit and the image recognition model in the system of 1, Figures 6 and 7 are views showing an example of the food image information photographed in the smart device.

As shown, the personalized nutrition and disease management system through the deep learning algorithm-based food image recognition of the present invention may include a smart device 100 as a user terminal and a nutrition management server 200 .

The smart device 100 communicates with the nutrition management server 200 as a portable terminal of a user capable of the Internet, and may be provided with an individual nutrition and disease management solution based on food and food recognition.

The smart device 100 may photograph food and food or transmit an existing photographed image to a server. For example, as illustrated in FIGS. 6 and 7 , the food image P10 photographed in the smart device 100 may be displayed, and at the bottom of the photographed food image P10, a hamburger 11, a taco ( 12), kebab 13, bulgogi burger 111, shrimp burger 121, chicken burger 131, and the like may be displayed.

The smart device 100 may receive image recognition information related to an image recognized from the nutrition management server 200 , and may receive and display image recognition information about a plant image photographed from the nutrition management server 200 .

The nutrition management server 200 may provide individual nutrition and disease management services based on food and food recognition at the request of the smart device 100, and the internal configuration of the server will be described in more detail below. .

The nutrition management server 200 includes a network interface 210 , a nutrition management unit 220 , a user-specific nutrition data database 230 , a food nutrition data database 240 , a food additive data database 250 , and an image It may include a recognition unit 260 and an image recognition model 270 .

The network interface 210 may communicate with the smart device 100 through a communication network to transmit/receive personalized nutrition and disease management information through deep learning algorithm-based food image recognition.

The nutrition management unit 220 may receive the food and the food image transmitted through the application, and transmit the image to the image recognition unit 260 .

The nutrition management unit 220 utilizes the food and nutrition DB 240 and the additive DB 250 based on the food and food list received from the image recognition model 270 to set the food and food name as the key to the existing data (food additive information and nutritional information) can be mapped.

The nutrition management unit 220 may store the user's intake record in the user nutrition DB 230 and transmit image recognition information related to the recognized image to the app client.

As illustrated in FIG. 2 , the nutrition management unit 220 may analyze and provide nutrition and ratio information for the cheeseburger and french fries recognized by the image recognition unit 260 . For example, the nutrition and ratio of cheeseburger are analyzed as carbohydrate 50.0g, protein 40.0g, and fat 20.0g, respectively, and the nutrition and ratio are analyzed as 0.3g carbohydrate, 4.0g protein, and 3.0g fat, respectively. can do.

The user-specific nutrition data database 230 may store user-specific nutrition data.

The food nutrition data database 240 may store food nutrition data.

The food additive data database 250 may store food additive data.

The image recognition unit 260 may receive food and food images, and may extract by designating a region of interest (ROI) of food and food from the image using image processing as illustrated in FIG. 5 .

The image recognition model 270 may recognize an image of food and food using a machine learning model learned based on a region of interest (ROI) of the image received from the image recognition unit 260 .

The image recognition model 270 can be trained using a convolutional neural network (CNN) as a Keras framework, which is a standard framework for neural network learning, and through this training, image analysis with food information You can create a learning model.

The image recognition model 270 may output the food and food information of the image from the deep learning model as a list, and may transmit the food and food image list extracted through the model to the nutrition management unit 220 .

The image recognition model 270 may recognize and provide a probability for food from the image recognized image as illustrated in FIG. 2 . For example, in the food image of FIG. 2 , cheeseburgers 85%, hamburgers 12%, tacos 3%, french fries 70%, cheese sticks 20%, and shrimp crackers 10% can be recognized.

The image recognition model 270 may receive image recognition information related to the recognized image from the Google electronic device, process it itself, or transmit it to the customer's mobile phone.

In particular, the image recognition method in the image recognition unit 260 and the image recognition model 270 described above, as illustrated in FIG. 3 , a deep learning technique used for object recognition may be applied. As a deep learning technique used for such object recognition, a convolutional neural network (CNN) may be used to perform image analysis.

For example, FIG. 3 shows an example of recognizing a banana image, which is a specific image, as a banana by a convolutional neural network (CNN).

That is, in the present invention, in order to train a deep network, a network architecture can be designed based on a large amount of labeled food data to learn based on a convolutional neural network (CNN) and design a model. The structure of a convolutional neural network (CNN) will be described in more detail with reference to FIG. 4 attached thereto.

As shown in FIG. 4, the structure of a convolutional neural network (CNN) can be composed of repetition of a convolution operation, an activation operation (mostly ReLU), and a pooling operation in the feature learning process, and features more than a certain number of times. After the learning process, the image can be converted into a one-dimensional vector through the flattening process.

That is, a convolutional neural network (CNN) is a type of neural network mainly used in speech recognition or image recognition. It is configured to process multi-dimensional array data, and is specialized in multi-dimensional array processing such as color images. Therefore, most of the techniques using deep learning in the image recognition field are based on convolutional neural networks (CNNs).

In the case of a general neural network, image data is processed as it is. In other words, since the entire image is considered as one data and accepted as an input, the characteristics of the image cannot be found, and the correct performance is not obtained when the position of the image is slightly changed or distorted. However, a convolutional neural network (CNN) can process an image by dividing it into several pieces instead of one piece of data. In this way, even if the image is distorted, it is possible to extract the partial characteristics of the image, so that the correct performance can be achieved.

In the case of a convolutional neural network (CNN), it can be created by combining several layers like the previous neural networks. However, unlike a general neural network, a convolutional layer and a pooling layer may be added.

One of the characteristics of a convolutional neural network (CNN) is that a convolutional layer and a pooling layer are added, but the 'Affine-ReLU' hierarchical structure used so far can be used in the layer close to the output. Also, in the last layer, the 'Affine-Softmax' combination can be used as it is.

In particular, the main task of convolution is to extract content feature points and may consist of one or more convolutional layers and one or more fully connected layers. Through convolution, the size of the content can be reduced and the noise of the content can be reduced.

ReLU represents a Rectified Linear Unit for nonlinear operation and the output is f(x)=max(0,x). There are other nonlinear functions such as tanh or sigmoid that can be used instead of ReLU. Because it is better, ReLU can be used.

The Pooling Layer section can reduce the number of parameters when the image is too large. Spatial pooling, also known as subsampling or downsampling, reduces the dimension of each map, but retains important information.

In fully connected mode, if the 2D image is directly attached to the input of the neural network without convolution and subsampling, the learning time may increase, the size of the network, and the number of variables may increase. By making it a dimensional matrix, we can map them all, one for each input end of the neural network.

You can then output the class and classify the image through the Softmax activation function.

8 is a diagram illustrating a personalized nutrition and disease management process through food image recognition between a user terminal and a nutrition management server, FIG. 9 is a diagram illustrating an example of Nutri-Score of a product by brand, and FIG. It is an example of an additive grade display screen, Figure 11 is a diagram showing an example of the top 10 food additive labels that should not be eaten, Figure 12 is a diagram showing an example of a nutritional component analysis for a specific food.

As shown, the nutrition management server 200 may receive a food image from the user terminal 100 (S11).

Then, the nutrition management server 200 may analyze the image according to the probability (S12).

Subsequently, the nutrition management server 200 may transmit food list information according to the image analysis result according to the probability to the user terminal 100 (S13).

Subsequently, the nutrition management server 200 may receive information on the food intake selected by the user from the user terminal 100 ( S14 ).

Subsequently, the nutrition management server 200 may set the food amount of the food selected by the user (S15).

Subsequently, the nutrition management server 200 may analyze (S16) information on food additives included in the food selected by the user.

Subsequently, the nutrition management server 200 may determine whether to mediate with respect to the food setting selected by the user based on the analyzed food additive information (S17).

Subsequently, the nutrition management server 200 may transmit a mediation message for the food setting selected by the user to the user terminal 100 according to the result of determining whether to mediate (S18). Here, the intervention message may be for providing customized diet information to the user based on the user nutritional information when the amount of food or food additives of the food selected by the user is inappropriate.

Then, the user terminal 100 may transmit the feedback information on the intervention message received from the nutrition management server 200 to the nutrition management server 200 (S19). Here, the feedback information for the intervention message is response information to the customized diet information provided from the nutrition management server 200 , and may include information that the user additionally requests for the customized diet information.

Subsequently, the nutrition management server 200 may store the food (food) intake history of the user according to the feedback information for the intervention message transmitted from the user terminal 100 (S20).

That is, the present invention may cause one or more processors to perform the following operations.

First, the nutrition management server 200 may receive food (food) information to check the risk information from the user.

Subsequently, the nutrition management server 200 may determine a grade based on a score calculation algorithm for food additive information of the Ministry of Food and Drug Safety and EWG grade contained in food corresponding to the received information.

Subsequently, the nutrition management server 200 may determine a grade based on a score calculation algorithm for Nutri-Score nutrition information corresponding to the received information.

Subsequently, the nutrition management server 200 may determine food (food) using a deep learning algorithm using big data, and transmit an item obtained above the previously learned similarity value to the client.

As described above, in the present invention, it is possible to receive food information for which risk information is to be confirmed from the user, analyze the user-specific nutrition database, food image data, and food additive data stored in the nutrition management server 200, and transmit it to the user device.

In other words, according to the present invention, an image including food (food) information may be acquired by a camera in the user device.

Accordingly, the nutrition management server 200 may determine the food included in the image obtained from the user device based on artificial intelligence.

Subsequently, the nutrition management server 200 may generate EWG food additive grade and Nutri-score grade information for food included in the image. In this case, when the type of food cannot be determined, the user selects an item with a predetermined high similarity value, transmits the selected item to the network interface, and determines that the retransmitted value is the food (food) selected by the end user.

For example, when the nutrition management server 200 cannot determine the type of food, at least one or more candidate foods (foods) are transmitted to the smart device 100 as illustrated in FIGS. 6 and 7 . Candidate group food (food) may be displayed on the user terminal.

Accordingly, the user can select any one food (food) on the displayed screen, and when the nutrition management server 200 recognizes that it is a hamburger but does not know the type of hamburger, 'Bulgogi Burger' or 'Shrimp' By providing information on at least one type of hamburger, such as 'burger' and 'chicken burger', the user selects any one hamburger among them, and the selected information may be transmitted to the nutrition management server 200 .

In the determining step, when the image information of food (food) includes a plurality of results, it is possible to extract a recognizable image, overlap the image, and transmit it to the network interface.

In addition, it may include a step in which the user can directly select the amount of food (food) intake between the step of determining and the step of generating the management information.

Also, it is possible to detect an edge region of the household items included in the image, and estimate the amount of food based on the size (length, area) of the household items.

For example, as shown in FIG. 9 , Nutri-Scores for at least one product by brand may be expressed as Nutriscore_A to Nutriscore_E for each product.

That is, it may be possible to perform each grading by analyzing food and food information based on the nutritional information labeling of EU, such as France and the UK.

In addition, the nutritional components such as calories, saturated fat, sugar, sodium, vitamins, dietary fiber, and protein of food and food are converted to each value based on 100 g, and the nutritional score is calculated and displayed by segmenting them into grades A to E. there is.

In addition, when explaining grade conversion with Nutri-Score, as in Tables 1 to 3 below, N component represents the sum of calories, saturated fat, sugar, and sodium score, and components constituting N are based on 100 g points can be assigned for each section.

P represents the sum of scores of vitamins, dietary fiber, and protein, and may be nutritional score = total N points - total P points.

Table 1 below shows the N grade calculation table, Table 2 shows the P grade calculation table, and Table 3 shows the Nutris grade calculation table.

[Table 1]

[Table 2]

[Table 3]

On the other hand, looking at the additive grade calculation of food, as shown in Table 4 below, the food additive grade can be calculated based on the EWG (The Environmental Working Group) grade, a non-profit organization in the United States.

[Table 4]

That is, No Concern and Lower Concern can be classified into A grade, Moderate can be classified into B grade, and Higher Concern can be classified into C grade, so that EWG-based food additive grade can be displayed visually in three stages.

For example, chemical additive grades for each food may be displayed as shown in FIG. 10, and in FIG. 11, by providing information to consumers based on the <Top 10 food additives that should not be eaten>, it is possible to effectively filter out dangerous food additives and consume and ingest them. can be presented so that

In addition, as shown in FIG. 12 , it is possible to provide nutritional component analysis information for a specific food. For example, food additives and nutritional components for cheeseburgers may be labeled as C and E grades, respectively. Nutritional components are divided into fat, carbohydrate, protein, calcium, and vitamin, and can be consumed in moderation or over/under intake, etc. can be displayed. In addition, information on sugar or sodium may be displayed so that it can be visually confirmed.

13 is a view showing another example of a method for managing personalized nutrition and disease through food image recognition, receiving individual body information and age, receiving environmental/chronic disease and allergy information, outputting personalized nutrition information, and then individual customization Food and food additive grades can be provided.

In particular, after receiving individual body information and age, it is possible to check health information by age group, food information selected by the user, food raw material information, food nutrition information, and food additive information.

In addition, after receiving environmental/chronic disease and allergy information, it is possible to check the information on the food selected by the user, the cooking method of the selected food, and information on the region, so that nutritional information tailored to each individual can be output.

As described above, although the present invention has been described with reference to the limited embodiments and drawings, the present invention is not limited thereto, and the technical idea of the present invention and the following by those of ordinary skill in the art to which the present invention pertains. Of course, various modifications and variations are possible within the scope of equivalents of the claims to be described.

100: smart device
200: Nutrition Management Server
210: network interface
220: nutrition management department
230: Nutritional data database by user
240: Food Nutrition Data Database
250: food additive data database
260: image recognition unit
270: image recognition model

Claims (4)

As a personalized nutrition and disease management method through food image recognition based on a deep learning algorithm in a nutrition management server,
Recognizing at least one food included in the food image information transmitted from the user terminal;
analyzing food additives and nutritional information for the recognized at least one food;
transmitting food additives and nutritional analysis results for the corresponding food to the user terminal; and
A personalized nutrition and disease management method through deep learning algorithm-based food image recognition comprising the step of transmitting, to the user terminal, customized diet information recommended based on pre-stored user nutritional information. The method according to claim 1,
Recognizing at least one food included in the food image information transmitted from the user terminal comprises:
designating and extracting a region of interest of at least one food included in the food image information transmitted from the user terminal; and
Personalized nutrition through food image recognition based on a deep learning algorithm, characterized in that it comprises the step of recognizing at least one or more foods included in food image information using a machine learning model learned based on the extracted region of interest and disease management methods. As a personalized nutrition and disease management system through food image recognition based on deep learning algorithms,
a user terminal for receiving nutrition and disease management services; and
Personalization through deep learning algorithm-based food image recognition including a nutrition management server that analyzes food image information transmitted from the user terminal and transmits customized diet information to the user terminal based on the analyzed result and user nutrition information Nutrition and disease management system. 4. The method according to claim 3,
The nutrition management server,
a network interface for communicating with the user terminal;
an image recognition unit for designating and extracting a region of interest of at least one food included in the food image information transmitted from the user terminal;
an image recognition model unit for recognizing at least one food included in food image information using a machine learning model learned based on the region of interest extracted from the image recognition unit; and
Deep learning algorithm-based, characterized in that it comprises a nutrition management unit that analyzes food additives and nutritional information for at least one food recognized by the image recognition model unit and recommends customized diet information based on pre-stored user nutritional information Personalized nutrition and disease management system through food image recognition.

Images