CN112053428A - Method and device for identifying nutritional information contained in food - Google Patents
Method and device for identifying nutritional information contained in food Download PDFInfo
- Publication number
- CN112053428A CN112053428A CN202010786491.2A CN202010786491A CN112053428A CN 112053428 A CN112053428 A CN 112053428A CN 202010786491 A CN202010786491 A CN 202010786491A CN 112053428 A CN112053428 A CN 112053428A
- Authority
- CN
- China
- Prior art keywords
- food
- dimensional model
- information
- volume
- nutrition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/68—Food, e.g. fruit or vegetables
Abstract
The invention discloses a method and a device for identifying nutritional information contained in food. The method comprises the following steps: firstly, collecting images of food from at least two different angles to obtain a multi-angle image of the food; then, carrying out three-dimensional model reconstruction on the food according to the multi-angle image to obtain a three-dimensional model which is 1:1 in proportion to the actual size of the food; then, performing three-dimensional model deconstruction processing on the three-dimensional model to obtain information such as food names, food volumes and food types; then, the information is associated and matched with data in a food nutrition information database established in advance to obtain nutrition information contained in food; and then displaying the nutritional information contained in the food. Thus, more accurate nutritional information or calorie information can be obtained. Moreover, compared with a mode of manually searching and calculating to obtain the nutrition information contained in the food, the mode is more convenient.
Description
Technical Field
The invention relates to the field of Augmented Reality (AR), in particular to a method and a device for reconstructing and identifying nutritional information contained in food by utilizing an AR technology.
Background
Modern people pay more and more attention to the quality of life, and particularly seek more and more reasonable matched nutritional formulas in the aspect of diet. For this reason, people need to know the nutritional components and calorie indexes of various foods.
In the conventional method, the nutrition information or calorie information of food is searched by inputting through a terminal, and further calculation is needed according to the weight or volume of the food to obtain more accurate nutrition information or calorie information. However, this process is very complicated and has poor practicability.
With the rise of AR technology in recent years, AR technology is also applied to the scene of identifying the nutritional information contained in food, but the following problems still exist in such a scheme: 1) the identification and calculation are mainly carried out from a single visual angle, and the single visual angle inevitably generates blind areas which cannot be covered, so that the problem that the obtained nutrition information and calorie information are inaccurate is easily caused. 2) The method cannot perform cross-scene identification on various foods, and cannot perform comparison of nutrition information or calorie information on the various foods on the same interface.
Disclosure of Invention
In view of the above problems, the present inventors have creatively provided a method and apparatus for identifying nutritional information contained in food.
According to a first aspect of embodiments of the present invention, there is provided a method of identifying nutritional information contained in a food, the method including: acquiring images of food from at least two different angles to obtain a multi-angle image of the food; carrying out three-dimensional model reconstruction on the food according to the multi-angle image to obtain a three-dimensional model which is 1:1 in proportion to the actual size of the food; performing three-dimensional model deconstruction processing on the three-dimensional model to obtain key information of food, wherein the key information of the food comprises at least one of a food name, a food material corresponding to the food, a food volume and a food material volume; performing correlation matching on the key information of the food and data in a pre-established food nutrition information database to obtain nutrition information contained in the food, wherein the data in the food nutrition information database comprises the composition relation of the food and food materials and nutrition information corresponding to the corresponding food materials in unit volume or unit mass; and displaying the nutritional information contained in the food.
According to an embodiment of the present invention, the food nutrition information database is located in a remote server, and before obtaining the nutrition information contained in the food by associating and matching the key information of the food with the data in the pre-established food nutrition information database, the method further includes: synchronizing a database of food nutrition information from a remote server to a local.
According to an embodiment of the present invention, after obtaining the nutritional information contained in the food by associating and matching the key information of the food with the data in the pre-established food nutritional information database, the method further includes: and associating the three-dimensional model with the nutritional information contained in the food and storing the three-dimensional model information in a food three-dimensional model information database.
According to an embodiment of the present invention, after displaying the nutritional information contained in the food, the method further comprises: displaying the food stored in the food three-dimensional model information database for the user to select; and comparing the nutrition information of at least two foods selected by the user and displaying the comparison result.
According to an embodiment of the present invention, a method for reconstructing a three-dimensional model of food according to a multi-angle image to obtain a three-dimensional model proportional to the actual size of the food by 1:1 includes: determining the area where the food is located from the multi-angle image and acquiring the shape and the outline of the food from the area; extracting characteristic points of the food from the multi-angle image according to the area where the food is located and the shape and the contour of the food; performing concentric multi-angle splicing processing on the feature points to obtain a three-dimensional model data file; model processing is performed according to the three-dimensional model data file to obtain a three-dimensional model in a ratio of 1:1 to the actual size of the food.
According to an embodiment of the present invention, the three-dimensional model deconstruction processing on the three-dimensional model to obtain the key information of the food includes: extracting the characteristics of the three-dimensional model to obtain at least one of color, texture and glossiness of food; obtaining food materials corresponding to the food according to at least one of color, texture and glossiness of the food; and further identifying and calculating the three-dimensional model to obtain the volume or the mass of the food material.
According to an embodiment of the present invention, further identifying and calculating the three-dimensional model to obtain the volume or the mass of the food material includes: selecting a first plane from the three-dimensional model as a datum plane; selecting an area where the food is located from the three-dimensional model on the reference plane; and calculating the volume of the area where the food material is located.
According to an embodiment of the present invention, after selecting an area where the food material is located, the method further includes: and performing at least one of contour recognition, flatness detection and perspective calculation on the area where the food material is located so as to correct the area where the food material is located.
According to an embodiment of the present invention, further identifying and calculating the three-dimensional model to obtain the volume or the mass of the food material includes: detecting whether the food is standard ratio food, if so, acquiring the volume or the mass of the food material from a pre-established standard ratio food database, wherein the standard ratio food database stores the food material corresponding to the standard ratio food and the volume or the mass of the food material.
According to a second aspect of embodiments of the present invention, there is provided an apparatus for identifying nutritional information contained in food, the apparatus including: the food processing device comprises an image acquisition module, a display module and a processing module, wherein the image acquisition module is used for acquiring images of food from at least two different angles to obtain multi-angle images of the food; the three-dimensional model reconstruction module is used for reconstructing a three-dimensional model of the food according to the multi-angle image to obtain a three-dimensional model which is 1:1 in proportion to the actual size of the food; the three-dimensional model deconstruction processing module is used for carrying out three-dimensional model deconstruction processing on the three-dimensional model to obtain key information of food, wherein the key information of the food comprises at least one of a food name, a food material corresponding to the food, a food volume and a food material volume; the nutrition information matching module is used for correlating and matching the key information of the food with data in a pre-established food nutrition information database to obtain nutrition information contained in the food, wherein the data in the food nutrition information database comprises the composition relation of the food and food materials and nutrition information corresponding to the corresponding food materials in unit volume or unit mass; and the nutritional information display module is used for displaying the nutritional information contained in the food.
According to an embodiment of the present invention, the apparatus further includes: and the database synchronization module is used for synchronizing the database of the food nutrition information from the remote server to the local.
According to an embodiment of the present invention, the apparatus further includes: and the three-dimensional model information database storage module is used for correlating the three-dimensional model with the nutritional information contained in the food and storing the three-dimensional model information into the food three-dimensional model information database.
According to an embodiment of the present invention, the apparatus further includes: the food selection module is used for displaying the food stored in the food three-dimensional model information database for the user to select; and the food nutrition information comparison module is used for comparing the nutrition information of at least two foods selected by the user and displaying the comparison result.
According to an embodiment of the present invention, a three-dimensional model reconstruction module includes: the food image range determining submodule is used for determining the area where the food is located from the multi-angle image and acquiring the shape and the outline of the food from the area; the characteristic point extraction submodule is used for extracting characteristic points of the food from the multi-angle image according to the area where the food is located and the shape and the contour of the food; the three-dimensional model data file acquisition submodule is used for carrying out concentric multi-angle splicing processing on the characteristic points to obtain a three-dimensional model data file; and the model processing submodule is used for carrying out model processing according to the three-dimensional model data file to obtain a three-dimensional model in a ratio of 1:1 to the actual size of the food.
According to an embodiment of the present invention, the three-dimensional model deconstruction processing module includes: the characteristic extraction submodule is used for extracting the characteristics of the three-dimensional model to obtain at least one of color, texture and glossiness of food; the food material identification submodule is used for acquiring food materials corresponding to the food according to at least one of color, texture and glossiness of the food; and the volume or mass calculation sub-module is used for further identifying and calculating the three-dimensional model to obtain the volume or mass of the food material.
According to an embodiment of the present invention, the volume or mass calculation submodule includes: the reference surface determining unit is used for selecting a first plane from the three-dimensional model as a reference surface; the food material area determining unit is used for selecting an area where food materials are located from the three-dimensional model on the reference surface; and the volume calculating unit is used for calculating the volume of the area where the food material is located.
According to an embodiment of the present invention, the volume or mass calculation submodule further includes: and the food material area correcting module is used for performing at least one of contour recognition, flatness detection and perspective calculation on the area where the food material is located so as to correct the area where the food material is located.
According to an embodiment of the present invention, the volume or mass calculating submodule is specifically configured to detect whether the food is a standard ratio food, and if so, obtain the volume or mass of the food material from a standard ratio food database established in advance, where the standard ratio food database stores the food material corresponding to the standard ratio food and the volume or mass of the food material.
The embodiment of the invention provides a method and a device for identifying nutritional information contained in food. The method comprises the following steps: firstly, collecting images of food from at least two different angles to obtain a multi-angle image of the food; then, carrying out three-dimensional model reconstruction on the food according to the multi-angle image to obtain a three-dimensional model which is 1:1 in proportion to the actual size of the food; then, performing three-dimensional model deconstruction processing on the three-dimensional model to obtain information such as food names, food volumes and food types; then, the information is associated and matched with data in a food nutrition information database established in advance to obtain nutrition information contained in food; and then displaying the nutritional information contained in the food. Because the embodiment of the invention acquires the food volume based on the multi-angle image reconstruction three-dimensional model and then identifies the food volume by acquiring the nutritional information contained in the food according to the corresponding relation between the food volume and the nutritional information, the accuracy of the acquired nutritional information or calorie information is higher; and compared with a mode of manually searching and calculating to obtain the nutritional information contained in the food, the mode is more convenient.
It is to be understood that the teachings of the present invention need not achieve all of the above advantages, but rather that specific embodiments may achieve specific technical effects, and that other embodiments of the present invention may achieve other advantages not mentioned above.
Drawings
The above and other objects, features and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description read in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which:
in the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.
FIG. 1 is a schematic flow chart illustrating the implementation of a method for identifying nutritional information contained in a food according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a data structure of a food nutrition information database according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating a method for identifying nutritional information contained in food according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a device for identifying nutritional information contained in food according to an embodiment of the present invention.
Detailed Description
In order to make the objects, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.
According to a first aspect of embodiments of the present invention, there is provided a method for identifying nutritional information contained in food, as shown in fig. 1, the method including: an operation 110 of acquiring images of a food from at least two different angles to obtain a multi-angle image of the food; operation 120, performing three-dimensional model reconstruction on the food according to the multi-angle image to obtain a three-dimensional model in a ratio of 1:1 to the actual size of the food; operation 130, performing three-dimensional model deconstruction processing on the three-dimensional model to obtain key information of the food, wherein the key information of the food includes at least one of a food name, a food material corresponding to the food, a food volume, and a food material volume; operation 140, performing association matching on the key information of the food and data in a pre-established food nutrition information database to obtain nutrition information contained in the food, wherein the data in the food nutrition information database comprises a composition relationship between the food and food materials and nutrition information corresponding to the corresponding food materials per unit volume or per unit mass; in operation 150, nutritional information contained in the food is displayed.
The nutritional information contained in the food mainly refers to at least one of data such as nutritional ingredients contained in the food, the mass of each nutritional ingredient, and calorie data of the food.
In operation 110, multi-angle images of food are mainly captured by cameras of various image capturing apparatuses. The various image acquisition devices comprise mobile terminal devices such as mobile phones, tablet computers and AR glasses. The multi-angle image refers to an image photographed from at least two different angles. In order to ensure that the user can acquire enough images shot at different angles, different shooting angle demonstration images or voice prompts can be provided for the user in actual application. After the acquisition of the multi-angle images is completed, the preliminary verification can be performed to ensure that the acquired multi-angle images of the food can be used for reconstructing a three-dimensional model.
The multi-angle image collected by the user is a data basis for performing subsequent steps such as three-dimensional model reconstruction, food volume calculation, nutrition information matching and the like, and the more comprehensive the angle of the collected image is, the clearer the image quality is, and the more accurate the finally obtained nutrition information is.
In operation 120, the three-dimensional model reconstruction refers to a process of reconstructing a three-dimensional model from a multi-angle image, and the method generally includes calibrating a camera, i.e., calculating a relationship between an image coordinate system of the camera and a world coordinate system, reconstructing three-dimensional information from information in a plurality of two-dimensional images, and constructing the three-dimensional model from the three-dimensional information. Specifically, the photos to be processed are extracted through various arrangements and features until textures and other functions are generated, so that three-dimensional model data files which can be used for three-dimensional reconstruction are obtained, and then three-dimensional reconstruction tools are used for generating three-dimensional models by using the three-dimensional model data files.
The embodiment of the invention is not limited to the architecture and algorithm used for reconstructing the three-dimensional model, and can be a 3D model reconstruction algorithm based on the combination of the Boosting architecture and Haar characteristics; the method can be a three-dimensional model reconstruction algorithm based on machine learning deep learning; the method can be a three-dimensional model reconstruction algorithm based on machine learning SVM or Bayes learning; may be a three-dimensional model reconstruction algorithm based on geometry analysis; or a texture-based food feature point model reconstruction algorithm. The implementer may select any suitable three-dimensional model reconstruction architecture or algorithm depending on the specific implementation conditions.
Through 3D model reconstruction of food, the composition of food materials of the food can be identified according to the analysis of the similarity, and the volume of the food materials can be accurately calculated, so that more accurate nutritional information can be obtained. The reconstruction of the three-dimensional model is also an important basis for subsequent steps such as food volume calculation, nutrition information matching and the like, and the higher the reduction degree of the three-dimensional model constructed at the position is, the more accurate the finally obtained nutrition information is.
In operation 130, the three-dimensional model deconstruction process is a process of extracting features of the three-dimensional model using image processing techniques and analyzing the features to identify food and obtain key information related to the food. The key information of the foods is information required for performing subsequent food nutrition information matching and nutrition information calculation, particularly food materials corresponding to the foods, and volume or quality information of the food materials.
In operation 140, a pre-established database of food nutrition information is used to store the composition relationship between food and food materials and the nutrition information corresponding to the corresponding food materials per unit volume or per unit mass. These nutritional information are reference information for identifying nutritional information contained in the food, and are the main basis for calculating the nutritional information contained in the food.
Usually, the food nutrition information database is created according to a certain data association relationship and information organization structure according to research results of nutriologists and related data accumulated in the industry.
Fig. 2 shows a schematic diagram of the data structure of the food nutrition information database. As shown in fig. 2, the food nutrition information database mainly stores the following data:
food names or food identifiers, such as "food a", "food B", and "food C", etc.;
food-corresponding ingredients, such as "ingredient 101", "ingredient 102", and "ingredient 103", etc., wherein each ingredient corresponds to a table of nutritional components and calorie data, in which nutritional information included in each unit mass of the ingredient is stored; in addition, each food material also corresponds to a mapping table of the unit volume and the unit mass of the food material, the table can be used for converting the volume of the food material into the mass, and then the nutritional information contained in the food material is calculated according to the nutritional ingredients and the calorie data table.
Wherein, the food material that food corresponds includes: cereals, potato starch, dried beans, vegetables, fungi and algae, fruits, nut seeds, livestock meat, poultry meat, milk, eggs, fish, shrimp, crab and shellfish, infant food, snack cookies, instant food, soft drink, alcoholic beverage, sugar candy, oil and fat, seasonings, medicine and food, and others. The nutritional composition and calorie data of food include: calories, protein, riboflavin, magnesium, fat, niacin, thiamine, calcium, iron, carbohydrates, vitamin C, manganese, dietary fiber, vitamin E, zinc, vitamin a, cholesterol, copper, carotene, potassium, phosphorus, retinol equivalents, sodium, selenium, and the like. The food material comprises fish-flavored shredded pork, ginger, garlic clove, soy sauce, refined salt, vinegar, pickled pepper, fresh soup, white sugar, water bean powder, Auricularia, carrot, bamboo shoot, and dried pepper.
Taking the food nutrition information database as an example, when the key information of food is associated and matched with the data in the food nutrition information database, the corresponding food material composition can be found according to the name of the food, and then the nutrition information of the unit mass of the corresponding food material can be obtained according to the food material composition. When no food material matched with the food name is formed, the food material of the food can be directly identified through an image identification technology, and then the nutrition information of the corresponding food material in unit mass is obtained. Then, the volume of each food material obtained according to the three-dimensional model structure is converted into the mass, and the mass is multiplied by the nutrition information of the unit mass of the food material, so that the nutrition information contained in each food material can be obtained. Then, the nutrition information of all food materials corresponding to the food is added to obtain the total nutrition information of the food. The nutritional information here is the nutritional information contained in the food to be identified by the embodiment of the present invention.
In operation 150, the display is typically made through a display screen of the user terminal. The user terminal can be a mobile terminal device such as a mobile phone, a tablet computer and AR glasses.
According to an embodiment of the present invention, the food nutrition information database is located in a remote server, and before obtaining the nutrition information contained in the food by associating and matching the key information of the food with the data in the pre-established food nutrition information database, the method further includes: synchronizing a database of food nutrition information from a remote server to a local.
In the embodiment, the food nutrition information database is located in a remote server and is synchronized to the local part when in use, and if the food nutrition information database is not used for a long time after the use is finished, the contents in the food nutrition information database can be cleared. Therefore, on one hand, the local storage space can be saved, on the other hand, the database access speed can be accelerated after the database is synchronized to the local area, so that the process of identifying the nutritional information contained in the food is quicker, and the remote database cannot become the performance bottleneck of the whole system due to overlarge access amount. Here, locally means a device for executing the method for identifying the nutritional information contained in the food according to the embodiment of the present invention, and is generally the mobile terminal device for capturing images and displaying the identification result.
According to an embodiment of the present invention, after obtaining the nutritional information contained in the food by associating and matching the key information of the food with the data in the pre-established food nutritional information database, the method further includes: and associating the three-dimensional model with the nutritional information contained in the food and storing the three-dimensional model information in a food three-dimensional model information database.
The three-dimensional model also has the characteristic of reusability, and the three-dimensional model and the nutritional information contained in food are associated and stored in the food three-dimensional model information database, so that the three-dimensional model can be conveniently called for reuse. Therefore, when a user wants to acquire the nutritional information contained in the food identified before again, the user does not need to acquire the multi-angle image again for identification, and the nutritional information contained in the food can be acquired again by inquiring the three-dimensional food model information database.
In addition, the food three-dimensional model information database can also be used for inquiring food nutrient components, namely, a user only needs to input a food name or select a related food model to inquire the content of the nutrient components such as food heat, fat, protein, vitamin A, carotene, calcium and the like, and the food three-dimensional model information database contains related food nutrient component knowledge, has complete data and is more convenient to inquire and call.
In addition, it should be noted that some intermediate files required for generating the three-dimensional model can be stored in the food three-dimensional model information database, and fine adjustment can be performed based on the existing intermediate files when a new model is generated, so that the reusability of information and data can be further improved.
According to an embodiment of the present invention, after displaying the nutritional information contained in the food, the method further comprises: displaying the food stored in the food three-dimensional model information database for the user to select; and comparing the nutrition information of at least two foods selected by the user and displaying the comparison result.
In some scenes, after the user identifies the nutrient elements and calories of the food, the user also wants to select other foods for comparison, and the nutrient information contained in the food is more in line with the current needs of the user. At the moment, the multi-nutrient component and calorie comparison of various foods can be completed by calling the existing information in the food three-dimensional model information database. For example, when the user selects existing food in the three-dimensional food model information database and compares the existing food with current food, the nutritional information contained in the selected food can be called from the three-dimensional food model information database and then compared with the nutritional information identified by the current food. In this way, the user may be helped to select a more appropriate food.
In addition, in the scene of augmented reality display, the embodiment can be used for repeatedly carrying out or directly comparing the nutritional information contained in various foods, and the three-dimensional model in the food three-dimensional model information database can be used for carrying out more visualized numerical display.
According to an embodiment of the present invention, a method for reconstructing a three-dimensional model of food according to a multi-angle image to obtain a three-dimensional model proportional to the actual size of the food by 1:1 includes: determining the area where the food is located from the multi-angle image and acquiring the shape and the outline of the food from the area; extracting characteristic points of the food from the multi-angle image according to the area where the food is located and the shape and the contour of the food; performing concentric multi-angle splicing processing on the feature points to obtain a three-dimensional model data file; model processing is performed according to the three-dimensional model data file to obtain a three-dimensional model in a ratio of 1:1 to the actual size of the food.
The method comprises the steps of determining an area where food is located from a multi-angle image, and obtaining the shape and the outline of the food from the area, wherein the similarity of the shape, the structure, the statistics, the texture, the environment, the height, the size and the like is mainly obtained by analyzing the similarity through an image recognition technology and an artificial vision technology.
The method for extracting the feature points may be a method based on a directional derivative, a method based on an image brightness contrast relationship, or a method based on mathematical morphology, and the specific method for extracting the feature points is not limited in this embodiment.
In the process of carrying out concentric multi-angle splicing processing on the feature points to obtain the three-dimensional model data file, a computer can be utilized to analyze and select features through spatial information and spectral information of different objects in the object image, the feature space is divided into subspaces which are not overlapped with each other, and then each pixel in the object image is classified into the subspaces. The three-dimensional model data file obtained through the process can be a file with a format of 3ds or other format files. When the three-dimensional model data file is displayed, point cloud data formed by the characteristic points can be visually seen.
In the present embodiment, in order to make the three-dimensional model obtained by the three-dimensional model reconstruction more accurate, the point cloud data in the model is also subjected to model processing after the three-dimensional model data file is obtained. The model processing here includes editing the point cloud data and rendering the three-dimensional model.
According to an embodiment of the present invention, the three-dimensional model deconstruction processing on the three-dimensional model to obtain the key information of the food includes: extracting the characteristics of the three-dimensional model to obtain at least one of color, texture and glossiness of food; obtaining food materials corresponding to the food according to at least one of color, texture and glossiness of the food; and further identifying and calculating the three-dimensional model to obtain the volume or the mass of the food material.
The characteristic extraction of the three-dimensional model is a main basis and a data basis for food material identification, and can effectively relieve the problem of dimension disaster frequently occurring in the field of pattern identification and play an important role in identification performance. In addition, the feature extraction of the three-dimensional model is important for similarity retrieval of the three-dimensional model, and a feature retrieval index is automatically established by matching the similarity between the model to be queried and the features of the target model, so that browsing and retrieval of a three-dimensional model database are realized.
The process of obtaining the food material corresponding to the food according to the characteristics is mainly to identify the food material according to the similarity degree of the shape, the structure, the statistics, the texture, the environment, the height, the size and the like, and the composition food material of the food can be fully deconstructed by the method. In addition, in the process of acquiring the food materials, the cooking method of the food, the name of the dish matched with the food and the like can be identified according to some characteristics on the image by using an artificial intelligence algorithm through a computer. The identification of the food materials and the raw material components mainly adopts image processing and computer vision as main methods, and the identification also lays an important foundation for effectively identifying the food volume in the follow-up process, namely, the volume and the quality of each food material can be fully solved by the method.
According to an embodiment of the present invention, further identifying and calculating the three-dimensional model to obtain the volume or the mass of the food material includes: selecting a first plane from the three-dimensional model as a datum plane; selecting an area where the food is located from the three-dimensional model on the reference plane; and calculating the volume of the area where the food material is located.
The volume calculation needs to use a certain plane in the three-dimensional model as a calculation reference plane, that is, the volume of the food above the reference plane is set and calculated. The computer then uses a computer program to calculate the volume of the selected region above this reference plane by selecting the region to be calculated. The reference plane determination may be set by means of parameter setting or a setpoint determination plane or the like. When the area needing to be calculated is selected, the area meeting parameter conditions can be automatically selected by inputting the altitude values, and the volume calculation of the area is carried out, wherein the parameter conditions are usually determined by the characteristics of a certain food material, and the area suspected to be the food material can be screened out through the parameter conditions. In addition, in order to simplify the calculation of the volume, a region with a fixed range can be set on the three-dimensional model, the three-dimensional model is automatically divided into a plurality of subareas according to different food materials, so that the volume calculation is carried out according to the food materials and the subareas, and then the volume of each food material can be obtained by classifying and summarizing.
According to an embodiment of the present invention, after selecting an area where the food material is located, the method further includes: and performing at least one of contour recognition, flatness detection and perspective calculation on the area where the food material is located so as to correct the area where the food material is located.
In order to make the volume calculation more accurate, in the embodiment, after the region where the food material is located is selected, fine recognition such as at least one of contour recognition, flatness detection, perspective calculation, and the like is further performed to correct the primarily recognized region, so that the error of the volume calculation is smaller.
According to an embodiment of the present invention, further identifying and calculating the three-dimensional model to obtain the volume or the mass of the food material includes: detecting whether the food is standard ratio food, if so, acquiring the volume or the mass of the food material from a pre-established standard ratio food database, wherein the standard ratio food database stores the food material corresponding to the standard ratio food and the volume or the mass of the food material.
In practical applications, there are also some foods with standard mix ratios, such as hamburgers, chips, rice packages, or cuisine with relatively fixed recipes sold by chain-operated food enterprises. These foods are usually prepared according to a specific food material ratio regulation or a recipe. In the present embodiment, the foods in the standard mix ratios and the specific food material mix ratio regulations or recipes corresponding to the foods are recorded in the standard mix ratio food database. For these foods, the volume calculation is not needed, and the food material of the food and the volume or mass of each food material can be obtained directly through food material proportioning regulation.
The detailed process from collecting the multi-angle images to identifying the nutritional information contained in the food by applying the method for identifying the nutritional information contained in the food according to the embodiment of the present invention will be described in detail with reference to fig. 3. The process mainly comprises the following steps:
3050, rendering to obtain a three-dimensional model;
3130, summarizing the volume or mass of each food material;
3170, prompting the user whether to select other foods for comparison, if so, continuing to step 3180, and if not, ending the execution;
It should be noted that fig. 3 is an application of the embodiment of the present invention, and is only an exemplary illustration, and does not limit the implementation manner of the embodiment of the present invention.
According to a second aspect of the embodiments of the present invention, there is provided an apparatus for identifying nutritional information contained in food, as shown in fig. 4, the apparatus 40 including: an image collecting module 401, configured to collect images of food from at least two different angles to obtain a multi-angle image of the food; a three-dimensional model reconstruction module 402, configured to perform three-dimensional model reconstruction on food according to the multi-angle image to obtain a three-dimensional model proportional to the actual size of the food by 1: 1; a three-dimensional model deconstruction processing module 403, configured to perform three-dimensional model deconstruction processing on the three-dimensional model to obtain key information of the food, where the key information of the food includes at least one of a food name, a food material corresponding to the food, a food volume, and a food material volume; a nutritional information matching module 404, configured to perform association matching on the key information of the food and data in a pre-established food nutritional information database to obtain nutritional information contained in the food, where the data in the food nutritional information database includes a composition relationship between the food and food materials and nutritional information corresponding to the corresponding food materials per unit volume or per unit mass; and a nutritional information display module 405 for displaying nutritional information contained in the food.
According to an embodiment of the present invention, the apparatus 40 further includes: and the database synchronization module is used for synchronizing the database of the food nutrition information from the remote server to the local.
According to an embodiment of the present invention, the apparatus 40 further includes: and the three-dimensional model information database storage module is used for correlating the three-dimensional model with the nutritional information contained in the food and storing the three-dimensional model information into the food three-dimensional model information database.
According to an embodiment of the present invention, the apparatus 40 further includes: the food selection module is used for displaying the food stored in the food three-dimensional model information database for the user to select; and the food nutrition information comparison module is used for comparing the nutrition information of at least two foods selected by the user and displaying the comparison result.
According to an embodiment of the present invention, the three-dimensional model reconstruction module 402 includes: the food image range determining submodule is used for determining the area where the food is located from the multi-angle image and acquiring the shape and the outline of the food from the area; the characteristic point extraction submodule is used for extracting characteristic points of the food from the multi-angle image according to the area where the food is located and the shape and the contour of the food; the three-dimensional model data file acquisition submodule is used for carrying out concentric multi-angle splicing processing on the characteristic points to obtain a three-dimensional model data file; and the model processing submodule is used for carrying out model processing according to the three-dimensional model data file to obtain a three-dimensional model in a ratio of 1:1 to the actual size of the food.
According to an embodiment of the present invention, the three-dimensional model deconstruction processing module 403 includes: the characteristic extraction submodule is used for extracting the characteristics of the three-dimensional model to obtain at least one of color, texture and glossiness of food; the food material identification submodule is used for acquiring food materials corresponding to the food according to at least one of color, texture and glossiness of the food; and the volume or mass calculation sub-module is used for further identifying and calculating the three-dimensional model to obtain the volume or mass of the food material.
According to an embodiment of the present invention, the volume or mass calculation submodule includes: the reference surface determining unit is used for selecting a first plane from the three-dimensional model as a reference surface; the food material area determining unit is used for selecting an area where food materials are located from the three-dimensional model on the reference surface; and the volume calculating unit is used for calculating the volume of the area where the food material is located.
According to an embodiment of the present invention, the volume or mass calculation submodule further includes: and the food material area correcting module is used for performing at least one of contour recognition, flatness detection and perspective calculation on the area where the food material is located so as to correct the area where the food material is located.
According to an embodiment of the present invention, the volume or mass calculating submodule is specifically configured to detect whether the food is a standard ratio food, and if so, obtain the volume or mass of the food material from a standard ratio food database established in advance, where the standard ratio food database stores the food material corresponding to the standard ratio food and the volume or mass of the food material.
Here, it should be noted that: the above description of the embodiment of the apparatus for identifying nutritional information contained in food is similar to the description of the embodiment of the method, and has similar beneficial effects to the embodiment of the method, and therefore, the detailed description is omitted. For technical details that have not been disclosed yet in the description of the embodiments of the apparatus for identifying nutritional information contained in food of the present invention, please refer to the description of the foregoing embodiments of the method of the present invention for understanding, and therefore, for brevity, will not be described again.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of a unit is only one logical function division, and there may be other division ways in actual implementation, such as: multiple units or components may be combined, or may be integrated into another device, or some features may be omitted, or not implemented. In addition, the coupling, direct coupling or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the devices or units may be electrical, mechanical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units; can be located in one place or distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, all the functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately regarded as one unit, or two or more units may be integrated into one unit; the integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
Those of ordinary skill in the art will understand that: all or part of the steps for realizing the method embodiments can be completed by hardware related to program instructions, the program can be stored in a computer readable storage medium, and the program executes the steps comprising the method embodiments when executed; and the aforementioned storage medium includes: various media capable of storing program codes, such as a removable storage medium, a Read Only Memory (ROM), a magnetic disk, and an optical disk.
Alternatively, the integrated unit of the present invention may be stored in a computer-readable storage medium if it is implemented in the form of a software functional module and sold or used as a separate product. Based on such understanding, the technical solutions of the embodiments of the present invention may be essentially implemented or a part contributing to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the methods of the embodiments of the present invention. And the aforementioned storage medium includes: a removable storage medium, a ROM, a magnetic disk, an optical disk, or the like, which can store the program code.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A method of identifying nutritional information contained in a food, the method comprising:
acquiring images of food from at least two different angles to obtain a multi-angle image of the food;
carrying out three-dimensional model reconstruction on the food according to the multi-angle image to obtain a three-dimensional model in a ratio of 1:1 with the actual size of the food;
performing three-dimensional model deconstruction processing on the three-dimensional model to obtain key information of the food, wherein the key information of the food comprises at least one of a food name, a food material corresponding to the food, a food volume and a food material volume;
performing correlation matching on the key information of the food and data in a pre-established food nutrition information database to obtain nutrition information contained in the food, wherein the data in the food nutrition information database comprises the composition relation of the food and food materials and nutrition information corresponding to the corresponding food materials in unit volume or unit mass;
and displaying the nutritional information contained in the food.
2. The method of claim 1, wherein the food nutrition information database is located in a remote server, and before the association matching of the key information of the food with the data in the pre-established food nutrition information database to obtain the nutrition information contained in the food, the method further comprises:
synchronizing the database of food nutrition information from the remote server to a local.
3. The method of claim 1, wherein after the matching of the key information of the food with the data in the pre-established database of food nutrition information to obtain the nutrition information contained in the food, the method further comprises:
and associating the three-dimensional model with the nutritional information contained in the food and storing the three-dimensional model information in a food three-dimensional model information database.
4. The method of claim 3, wherein after said displaying nutritional information contained in said food, said method further comprises:
displaying the food stored in the food three-dimensional model information database for the user to select;
and comparing the nutrition information of at least two foods selected by the user and displaying the comparison result.
5. The method of claim 1, wherein the three-dimensional model reconstruction of the food from the multi-angle image to obtain a three-dimensional model with a ratio of 1:1 to an actual size of the food comprises:
determining the area where the food is located from the multi-angle image and acquiring the shape and the outline of the food from the area;
extracting feature points of the food from the multi-angle image according to the area where the food is located and the shape and the contour of the food;
performing concentric multi-angle splicing processing on the feature points to obtain a three-dimensional model data file;
and performing model processing according to the three-dimensional model data file to obtain a three-dimensional model in a ratio of 1:1 to the actual size of the food.
6. The method of claim 1, wherein the performing three-dimensional model deconstruction processing on the three-dimensional model to obtain key information of the food comprises:
extracting the characteristics of the three-dimensional model to obtain at least one of color, texture and glossiness of the food;
acquiring food materials corresponding to the food according to at least one of the color, the texture and the glossiness of the food;
and further identifying and calculating the three-dimensional model to obtain the volume or the mass of the food material.
7. The method of claim 6, wherein said further identifying and calculating said three-dimensional model to obtain a volume or mass of said food material comprises:
selecting a first plane from the three-dimensional model as a datum plane;
selecting an area where food is located from the three-dimensional model above the reference plane;
and calculating the volume of the area where the food material is located.
8. The method of claim 7, wherein after the selecting the area in which the food material is located, the method further comprises:
and performing at least one of contour recognition, flatness detection and perspective calculation on the region where the food material is located so as to correct the region where the food material is located.
9. The method of claim 6, wherein said further identifying and calculating said three-dimensional model to obtain a volume or mass of said food material comprises:
detecting whether the food is standard ratio food, if so, obtaining the volume or the mass of the food material from a pre-established standard ratio food database, wherein the standard ratio food database stores the food material corresponding to the standard ratio food and the volume or the mass of the food material.
10. An apparatus for identifying nutritional information contained in a food, the apparatus comprising:
the food processing device comprises an image acquisition module, a display module and a processing module, wherein the image acquisition module is used for acquiring images of food from at least two different angles to obtain multi-angle images of the food;
the three-dimensional model reconstruction module is used for performing three-dimensional model reconstruction on the food according to the multi-angle image to obtain a three-dimensional model in a ratio of 1:1 with the actual size of the food;
the three-dimensional model deconstruction processing module is used for performing three-dimensional model deconstruction processing on the three-dimensional model to obtain key information of the food, wherein the key information of the food comprises at least one of a food name, a food material corresponding to the food, a food volume and a food material volume;
the nutrition information matching module is used for correlating and matching the key information of the food with data in a pre-established food nutrition information database to obtain nutrition information contained in the food, wherein the data in the food nutrition information database comprises the composition relation between the food and food materials and nutrition information corresponding to the corresponding food materials in unit volume or unit mass;
and the nutritional information display module is used for displaying the nutritional information contained in the food.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010786491.2A CN112053428A (en) | 2020-08-07 | 2020-08-07 | Method and device for identifying nutritional information contained in food |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010786491.2A CN112053428A (en) | 2020-08-07 | 2020-08-07 | Method and device for identifying nutritional information contained in food |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112053428A true CN112053428A (en) | 2020-12-08 |
Family
ID=73601455
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010786491.2A Pending CN112053428A (en) | 2020-08-07 | 2020-08-07 | Method and device for identifying nutritional information contained in food |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112053428A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112632935A (en) * | 2020-12-23 | 2021-04-09 | 广州城市职业学院 | Food nutrition label generation method, computer device and storage medium |
| CN113496488A (en) * | 2021-07-16 | 2021-10-12 | 深圳市乐福衡器有限公司 | Method and system for acquiring nutrition information, shooting terminal and storage medium |
| US20220222844A1 (en) * | 2021-01-13 | 2022-07-14 | Nuvilabs Co., Ltd. | Method, device, and program for measuring food |
| CN116825286A (en) * | 2023-08-31 | 2023-09-29 | 北京四海汇智科技有限公司 | Food ingredient identification and nutrition recommendation system |
| CN116884571A (en) * | 2023-09-07 | 2023-10-13 | 北京四海汇智科技有限公司 | Meal weight intelligent evaluation system based on image processing |
-
2020
- 2020-08-07 CN CN202010786491.2A patent/CN112053428A/en active Pending
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112632935A (en) * | 2020-12-23 | 2021-04-09 | 广州城市职业学院 | Food nutrition label generation method, computer device and storage medium |
| CN112632935B (en) * | 2020-12-23 | 2024-02-09 | 广州城市职业学院 | Food nutrition label generating method, computer device and storage medium |
| US20220222844A1 (en) * | 2021-01-13 | 2022-07-14 | Nuvilabs Co., Ltd. | Method, device, and program for measuring food |
| CN113496488A (en) * | 2021-07-16 | 2021-10-12 | 深圳市乐福衡器有限公司 | Method and system for acquiring nutrition information, shooting terminal and storage medium |
| CN116825286A (en) * | 2023-08-31 | 2023-09-29 | 北京四海汇智科技有限公司 | Food ingredient identification and nutrition recommendation system |
| CN116825286B (en) * | 2023-08-31 | 2023-11-14 | 北京四海汇智科技有限公司 | Food ingredient identification and nutrition recommendation system |
| CN116884571A (en) * | 2023-09-07 | 2023-10-13 | 北京四海汇智科技有限公司 | Meal weight intelligent evaluation system based on image processing |
| CN116884571B (en) * | 2023-09-07 | 2023-12-12 | 北京四海汇智科技有限公司 | Meal weight intelligent evaluation system based on image processing |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112053428A (en) | Method and device for identifying nutritional information contained in food | |
| CN107873101B (en) | Imaging system for object recognition and evaluation | |
| Chen et al. | Automatic chinese food identification and quantity estimation | |
| Beijbom et al. | Menu-match: Restaurant-specific food logging from images | |
| US9734426B2 (en) | Automated food recognition and nutritional estimation with a personal mobile electronic device | |
| Miyazaki et al. | Image-based calorie content estimation for dietary assessment | |
| Pouladzadeh et al. | Using graph cut segmentation for food calorie measurement | |
| US11823042B2 (en) | System for measuring food weight | |
| Kitamura et al. | Image processing based approach to food balance analysis for personal food logging | |
| JP2010061383A (en) | Automatic recipe creation apparatus and program | |
| US10331953B2 (en) | Image processing apparatus | |
| JP2010061381A (en) | Automatic recipe preparing device and program | |
| CN110969047A (en) | Method and device for identifying food materials and refrigerator | |
| Sudo et al. | Estimating nutritional value from food images based on semantic segmentation | |
| CN110366731B (en) | Systems, methods, and storage media for image capture for instruction meal | |
| CN111522848A (en) | Method, device, computer equipment and storage medium for processing dish data | |
| CN112464013A (en) | Information pushing method and device, electronic equipment and storage medium | |
| Lee et al. | Aifood: a large scale food images dataset for ingredient recognition | |
| Akpa et al. | Smartphone-based food weight and calorie estimation method for effective food journaling | |
| CN110675389A (en) | Food recommendation method, storage medium and intelligent household equipment | |
| JP2020135417A (en) | Information device, program and method for estimating usage of food or seasoning | |
| JP2018049584A (en) | Meal size estimation program, meal size estimation method, and meal size estimation apparatus | |
| CN112966617A (en) | Method for generating balance image, method and device for training image generation model | |
| CN114641830A (en) | Food volume and calorie calculation method, electronic device, electronic apparatus, and storage medium | |
| Sombutkaew et al. | Image-based Thai Food Recognition and Calorie Estimation using Machine Learning Techniques |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |