KR20210092552A - Method and device for detecting nutrition data of product - Google Patents

Abstract

A method for extracting nutrition information of a product comprises: a step of obtaining an image of a product; a step of recognizing a label of the product by inputting the image of the product in a first deep learning model; a step of recognizing a content information object comprising content information by inputting the label image of the product in a second deep learning model; and a step of recognizing content information from the content information object through preset recognition means. According to the present invention, a user can correctly use products according to component and content information.

Description

METHOD AND DEVICE FOR DETECTING NUTRITION DATA OF PRODUCT

The present invention relates to a method and apparatus for extracting nutritional information of a product.

In our daily life, everyone faces a situation in which they purchase and use various products in various situations. Buyers generally consider various factors such as design, price, and practicality when choosing a product. However, there are special products that require checking the detailed components of the product. These include health supplements, pharmaceuticals, and cosmetics. These products are directly related to health and require special attention, but it is difficult for general users to check all the constituent substances. In this state, there is a risk of misuse and abuse in the process of continuing use indiscriminately.

In order to recognize the ingredients of such products, various object detection techniques may be used. The object detection technique refers to a technique for identifying where (x, y) and in which size (w, h) an object (label) to be detected exists in an image. A general object detection technique detects an object in an image and displays which object it is in the form of a bounding box.

Mainly used object detection techniques include R-CNN (Convolutional Neural Network) and Fast/Faster R-CNN. R-CNN is a technology that extracts input images into about 2,000 sub-images, classifies them through CNN, and then classifies each object through SVM (Support Vector Machine).

Fast/Faster R-CNN is a technology that uses RPN (Region Proposal Networks) to pass through the entire image through the CNN at once without all proposals going through the CNN in order to improve the bottleneck of R-CNN.

Prior art literature: Korea Patent Publication No. 2018-0065856

An object of the present invention is to provide a method and apparatus for acquiring an image of a product and inputting it into the first deep learning model to recognize the label of the product and extract nutritional information of the product.

Recognizing the content information object including the content information by inputting the image of the label of the recognized product into the second deep learning model, and extracting the nutritional information of the product by recognizing the content information from the content information object through a preset recognition means A method and apparatus are provided.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention includes the steps of acquiring an image of a product, inputting the image of the product into a first deep learning model to recognize the label of the product, the Product nutrition information, comprising: inputting an image of a product label into a second deep learning model to recognize a content information object including content information; and recognizing content information from the content information object through a preset recognition means An extraction method may be provided.

Another embodiment of the present invention is an acquisition unit for acquiring an image of a product, a label recognition unit for recognizing a label of the product by inputting the image of the product into a first deep learning model, and a second image of the label of the product Provided is a product nutrition information extraction device comprising an object recognition unit for recognizing a content information object including content information by input into a deep learning model, and a content information recognition unit for recognizing content information from the content information object through a preset recognition means can do.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, the label of the product is recognized from the image of the product, the nutrition information including the ingredient information and the content information of the product is extracted from the image of the label of the product, and the extracted nutrition A method and apparatus for providing information to a user may be provided.

By displaying the ingredient information and content information of the product through the application, it is possible to provide a method and apparatus for extracting nutritional information of the product so that the user can use the product according to the correct dosage method.

1 is a block diagram of an apparatus for extracting product nutrition information according to an embodiment of the present invention.
2 is an exemplary view illustrating a label of a product recognized from an image of the product according to an embodiment of the present invention.
3 is an exemplary diagram illustrating an object of ingredient name and content information of a product recognized from an image of a label of the product according to an embodiment of the present invention.
4 is an exemplary view showing a screen for providing information on the ingredient name and content of a product according to an embodiment of the present invention.
5 is a flowchart of a method of extracting nutritional information of a product in the apparatus for extracting nutritional information of a product according to an embodiment of the present invention.
6 is an exemplary view comparing a method for extracting product nutrition information between the present invention and the prior art according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in the present specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an apparatus for extracting product nutrition information according to an embodiment of the present invention. Referring to FIG. 1 , the product nutrition information extraction device 100 includes an acquisition unit 110 , a label recognition unit 120 , an object recognition unit 130 , a content information recognition unit 140 , and a mapping unit 150 . can do.

The acquisition unit 110 may acquire an image of the product. For example, the acquiring unit 110 may acquire an image obtained by taking a close-up shot of a product in real time.

The label recognition unit 120 may recognize the label of the product by inputting the image of the product into the first deep learning model. Here, the first deep learning model may be a You Only Look Once (YOLO) network. For example, the label recognition unit 120 may recognize the label of the product by inputting the image of the product into the first deep learning model.

Unlike the existing R-CNN and Fast/Faster R-CNN, the YOLO network applies one neural network to the entire image, divides the image into regions, predicts the bounding box and probability of each region, and reports the entire image. It is a method of providing prediction information. However, since the YOLO network is specialized for real-time object recognition, it has a disadvantage in that it does not perform well in the process of extracting text from an image.

In the present invention, by using the YOLO network to track the object in multiple steps, it is possible to provide a method for extracting product nutrition information that utilizes the YOLO network and exhibits good performance.

The label recognition unit 120 includes a bounding box indicating the position of an object in an image through the YOLO network, and class probability, which is a probability indicating which set among predefined sets each bounding box is close to. ) can be explored.

By using the YOLO network, the label recognition unit 120 regards object recognition as a single regression problem to determine the relationship between one independent variable and one dependent variable, so that the It is possible to guess the type and location.

The process of recognizing the label of the product in the label recognition unit 120 will be described in detail with reference to FIG. 2 .

2 is an exemplary view illustrating a label of a product recognized from an image of the product according to an embodiment of the present invention. 2, when the label recognition unit 120 inputs the image 200 of the product to the first deep learning model, the first deep learning model divides the image 200 of the product in a grid form, and Reliability of each corresponding bounding box may be calculated.

Thereafter, the label recognition unit 120 determines the final bounding box with the highest reliability based on the reliability of a plurality of bounding boxes through a single convolutional network of the first deep learning model to label the product. (250) can be recognized. Here, it can be confirmed that the label 250 of the product is recognized through the bounding box marked 'Label'.

Through this process, the label recognition unit 120 may recognize the label of the product, and finally extract only the image (the image corresponding to the bounding box) of the label 250 of the product to generate the resulting image.

Returning to FIG. 1 again, the object recognition unit 130 independently performs the object recognition process twice using the second deep learning model for the image of the label of the product, and the ingredient name object and content information from the image of the label of the product Each object can be recognized. Here, the second deep learning model may also be a YOLO network. In the present invention, the accuracy of object recognition can be further improved by independently performing two object recognition processes using the second deep learning model. The process of recognizing the ingredient name object and the content information object will be described in detail with reference to FIG. 3 .

3 is an exemplary diagram illustrating an object of ingredient name and content information of a product recognized from an image of a label of the product according to an embodiment of the present invention. Referring to FIG. 3 , the object recognition unit 130 may recognize the ingredient name object including the ingredient name by inputting the image 300 of the label of the product into the second deep learning model. Here, the component name object may be recognized through the second deep learning model from which various component name images have been learned.

For example, the object recognition unit 130 divides the image 300 of the label of the product into a grid form, calculates the reliability of each detailed bounding box corresponding to the grid, and based on the reliability of a plurality of detailed bounding boxes and "Calories" (310), "Total Fat" (320), "Cholesterol" (330), "Sodium" (340) and "Total Carbohydrate" (350) In the same way, the component name object can be recognized.

The object recognition unit 130 may recognize the content information object including the content information by inputting the image 300 of the label of the product into the second deep learning model. Here, the second deep learning model can recognize an object having the same shape by learning the content information expressed in the form of 'number%'.

For example, the object recognition unit 130 divides the image 300 of the label of the product into a grid form, calculates the reliability of each detailed bounding box corresponding to the grid, and based on the reliability of a plurality of detailed bounding boxes Thus, by determining the final bounding box with the highest reliability, "content" including the "%" character can be recognized as a content information object. In this case, the content information objects corresponding to "content" may be extracted and learned from a separate image.

As such, the reason for using the YOLO network for both the first deep learning model and the second deep learning model is to increase the accuracy of performing the YOLO network. The present invention primarily allows the YOLO network to focus only on recognizing product labels through the first deep learning model, thereby improving the accuracy of label recognition including nutritional information of products within the entire image. Thereafter, the second deep learning model allows the YOLO network to focus only on the ingredient name object of the product and the content information object of the product, thereby improving speed and accuracy compared to searching for ingredient names from the entire image.

Thereafter, the content information recognition unit 140 may recognize the content information from the content information object through a preset recognition means. Here, the preset recognition means may be an optical character reader (OCR). For example, the content information recognizing unit 140 may perform "0%" (325), "0%" (335), "1%" (345), and "2%" (355) from the content information object through OCR. It is possible to recognize the number corresponding to the content information as shown.

The reason for recognizing only content information using OCR in the present invention is that, as in the prior art, when all texts included in the entire image are to be recognized using OCR, the larger the image, the more the image content, the slower the speed and the lower the accuracy. Because. Therefore, in the present invention, by recognizing only the content information object detected through the multi-step YOLO network through OCR, it is possible to process the nutritional information of the product in real time at an improved speed.

The mapping unit 150 may map the ingredient name included in the recognized ingredient name object and the content information included in the recognized content information object.

The mapping unit 150 may map content information corresponding to each ingredient name based on the stored location information of each ingredient name object and each content information object. Here, the mapping unit 150 compares the y value of the coordinates corresponding to the ingredient name object and the y value of the coordinates corresponding to the content information object, and then forms a pair of the ingredient name and the content information located at a similar position to be mapped. .

For example, referring to FIG. 3 , the mapping unit 150 maps the component object corresponding to "Total Fat" 320 with "0%" (325) having a similar y-coordinate value among the content information objects, The component object corresponding to "Total Carbohydrate" 350 may be mapped to "2%" (355) having a similar y-coordinate value among content information objects.

However, in the case of the component object corresponding to the “Calories” 310 , the mapping unit 150 may not perform mapping because there is no content information object having a similar y-coordinate value.

Through this process, the product nutritional information extraction apparatus 100 may finally recognize ingredient names such as "Total Fat", "Saturated Fat", and "Cholesterol" and content information mapped thereto.

4 is an exemplary view showing a screen for providing information on the ingredient name and content of a product according to an embodiment of the present invention. Referring to FIG. 4 , the product nutrition information extraction device 100 takes an image of a product through an application that provides nutritional information of the product, recognizes the image of the label of the product in real time, and then the ingredient name object and content of the product By recognizing the information object and mapping the ingredient name included in the ingredient name object and the content information included in the content information object, it is possible to provide the ingredient name of the product and content information of the corresponding ingredient.

For example, the product nutrition information providing device 100 may display “Calories: 30%”, “Total Fat: 0%”, “Cholesterol: 0%”, and “Sodium: 1” through the product recognition result screen 400 . %", "Total Carbohydrate: 2%", along with the ingredient name, content information corresponding to the ingredient can be provided.

The product nutritional information extraction apparatus 100 may be executed by a computer program stored in a medium including a sequence of instructions for extracting nutritional information of a product. When the computer program is executed by the computing device, it acquires the image of the product, inputs the image of the product into the first deep learning model to recognize the label of the product, and inputs the image of the label of the product into the second deep learning model to and a sequence of commands for recognizing a content information object including content information and recognizing content information from the content information object through a preset recognition means.

5 is a flowchart of a method of extracting nutritional information of a product in the apparatus for extracting nutritional information of a product according to an embodiment of the present invention. The method of extracting nutritional information of a product in the product nutritional information extraction apparatus 100 illustrated in FIG. 5 includes steps of time-series processing according to the embodiments illustrated in FIGS. 1 to 4 . Therefore, even if omitted below, it is also applied to the method of extracting nutritional information of a product by the product nutritional information extraction apparatus 100 according to the embodiment shown in FIGS. 1 to 4 .

In step S510, the product nutrition information extraction apparatus 100 may acquire an image of the product.

In step S520, the product nutrition information extraction apparatus 100 may recognize the label of the product by inputting the image of the product into the first deep learning model.

In step S530, the product nutrition information extracting apparatus 100 may recognize the ingredient name object including the ingredient name by inputting the image of the label of the product into the second deep learning model.

In step S540, the product nutrition information extraction apparatus 100 may recognize the content information object including the content information by inputting the image of the label of the product into the second deep learning model.

In step S550, the product nutrition information extraction apparatus 100 may recognize the content information from the content information object through a preset recognition means.

In step S560, the product nutrition information extraction apparatus 100 may map the ingredient name included in the recognized ingredient name object and the content information included in the recognized content information object.

In the above description, steps S510 to S560 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted if necessary, and the order between the steps may be switched.

6 is an exemplary view comparing a method for extracting product nutrition information between the present invention and the prior art according to an embodiment of the present invention. Here, the prior art uses only OCR for label recognition of products (Grubert, O., Gao, L., "recognition of nutrition facts, labels from mobile images", Technical Report, Stanford University, Apr. 2014.)

6, the comparison items between the present invention 600 and the prior art 610 are a real-time information extraction item 620, an OCR item 621, an object recognition item 622, a multi-level object recognition item 623, and It consists of target video items (624).

The present invention 600 extracts the ingredient name and content information for each ingredient in real time, uses OCR as a preset recognition means, recognizes the label of the product, recognizes the ingredient object and the content information object, respectively, and YOLO Objects are used in multiple steps using the first deep learning model and the second deep learning model composed of a network, and real-time images are targeted.

On the other hand, it can be seen that the prior art 610 shows a lot of difference in that the nutritional information of the product is recognized by using the OCR recognition technology from the image image taken previously. If you want to understand the nutritional information of a product using only OCR recognition technology, it takes a lot of time as a large amount of data cannot be processed in real time through OCR. There were drawbacks to not being able to.

Therefore, in the present invention, by recognizing the label of the product in real time and recognizing the component object corresponding to the ingredient name in the image of the product label, the OCR technique does not recognize the entire label, but only the character corresponding to the content information. By recognizing it, it is possible to provide nutritional information of the product in real time.

The method for extracting nutritional information of a product from the product nutritional information extraction apparatus described through FIGS. 1 to 6 is also in the form of a computer program stored in a medium executed by a computer or a recording medium including instructions executable by the computer. can be implemented. In addition, the method for extracting nutritional information of a product in the product nutritional information extraction apparatus described with reference to FIGS. 1 to 6 may be implemented in the form of a computer program stored in a medium executed by a computer.

Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The description of the present invention described above is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: product nutritional information extraction device
110: acquisition unit
120: label recognition unit
130: object recognition unit
140: content information recognition unit
150: mapping unit

Claims (10)

A method for extracting nutritional information of a product, the method comprising:
acquiring an image of the product;
Recognizing the label of the product by inputting the image of the product into a first deep learning model;
Recognizing a content information object including content information by inputting the image of the label of the product into a second deep learning model; and
Recognizing content information from the content information object through a preset recognition means
A method of extracting product nutritional information that includes.
The method of claim 1,
Recognizing the content information object including the content information by inputting the image of the label of the product into the second deep learning model
Recognizing an ingredient name object including an ingredient name by inputting the image of the label of the product into a second deep learning model
A method of extracting product nutritional information that includes.
3. The method of claim 2,
Mapping the ingredient name included in the recognized ingredient name object and the content information included in the recognized content information object
Which will further include, product nutritional information extraction method.
The method of claim 1,
The predetermined recognition means is OCR (Optical Character Reader), the product nutritional information extraction method.
The method of claim 1,
The first deep learning model and the second deep learning model is a YOLO (You Only Look Once) network, the product nutrition information extraction method.
A device for extracting nutritional information of a product, comprising:
an acquisition unit for acquiring an image of the product;
a label recognition unit for recognizing a label of the product by inputting the image of the product into a first deep learning model;
an object recognition unit for recognizing a content information object including content information by inputting an image of the label of the product into a second deep learning model; and
Content information recognition unit for recognizing content information from the content information object through a preset recognition means
That comprising a, product nutritional information extraction device.
7. The method of claim 6,
The object recognition unit inputting the image of the label of the product into a second deep learning model to recognize the ingredient name object including the ingredient name, product nutrition information extraction device.
8. The method of claim 7,
The object recognition unit will map the ingredient name included in the recognized ingredient name object and the content information included in the recognized content information object, product nutrition information extraction device.
7. The method of claim 6,
The preset recognition means is OCR (Optical Character Reader), the product nutritional information extraction device.
7. The method of claim 6,
The first deep learning model and the second deep learning model is a YOLO (You Only Look Once) network, the product nutritional information extraction device.

Images