CN107477971B

CN107477971B - Method and equipment for managing food in refrigerator

Info

Publication number: CN107477971B
Application number: CN201710658671.0A
Authority: CN
Inventors: 葛莹; 章佳; 李文轩; 史志建
Original assignee: Samsung Electronics China R&D Center; Samsung Electronics Co Ltd
Current assignee: Samsung Electronics China R&D Center; Samsung Electronics Co Ltd
Priority date: 2017-08-04
Filing date: 2017-08-04
Publication date: 2020-09-18
Anticipated expiration: 2037-08-04
Also published as: CN107477971A

Abstract

The application discloses a method for managing food in a refrigerator, which comprises the following steps: identifying the type and the quantity of food put into the refrigerator according to an image shot by an external camera of the refrigerator; determining the position of food put into the refrigerator according to an image shot by an internal camera of the refrigerator; generating an AR label for the food put into the refrigerator, and recording the information of the food; after receiving a food inquiry request of a user, displaying an AR label of the inquired food; wherein the information of the food at least comprises a food category, a number and a location; and after receiving an image viewing request of a user, displaying the image shot by the internal camera in real time. By the application, food in the refrigerator can be managed more efficiently.

Description

Method and equipment for managing food in refrigerator

Technical Field

The application relates to a refrigerator technology, in particular to a method and equipment for managing food in a refrigerator.

Background

In the prior art of refrigerators, refrigerator food material management mainly comprises food identification, food information input, food placement area allocation and menu recommendation.

In the existing panoramic image technology, how to identify and how to obtain images at different positions through a camera is mainly used to form a panoramic image.

Existing AR technologies are currently in the beginning stage and are very widely used, including: AR head-mounted devices, social systems, picture video presentation systems, scene customization, games & education, photography, tour guides, smart tables, AR tags, AR books, AR processing, etc.

The existing dual-camera/multi-camera technology is mainly used for image shooting, and the realizable functions include increasing shooting visual angle, depth of field, image scanning, space positioning, building a three-dimensional model, controlling zooming, image correction, ranging, measuring length and the like.

The current refrigerator technology has the following problems:

1. the food material checking function of the existing refrigerator can only see a single plane image, and blocked food is difficult to see clearly;

2. after the food in the existing refrigerator is taken away, the state of the shelf life label is not updated;

3. food searching of the existing refrigerator can only be carried out through photos and lists, and how many foods are overdue quickly is not intuitive;

4. the existing refrigerator can not identify and distinguish the same food put in different time, and has no prompt for the food which is not suitable to be put together;

5. the image identification of the existing refrigerator food is to compare and analyze the front and back of a plurality of pictures or intercept video clips, and the pictures need to be taken again before observing things every time, which is inconvenient for users to observe and interact.

Disclosure of Invention

The application provides a management method for food in a refrigerator, which can more effectively realize the management of the food.

In order to achieve the purpose, the following technical scheme is adopted in the application:

a method of managing food in a refrigerator, comprising:

identifying at least the category and the quantity of food put into the refrigerator according to an image shot by an external camera of the refrigerator;

determining the position of food put into the refrigerator according to an image shot by an internal camera of the refrigerator;

generating an AR label for the food put into the refrigerator, and recording the information of the food; after receiving a food inquiry request of a user, displaying an AR label of the inquired food; wherein the information of the food at least comprises a food category, a number and a location;

and after receiving an image viewing request of a user, displaying the image shot by the internal camera in real time.

Preferably, the number of the internal cameras is at least two, and the positions of different internal cameras are different;

the position of the food placed in the refrigerator is a three-dimensional space position determined according to images shot by all internal cameras;

after the tag is generated, generating an AR panorama according to images shot by all internal cameras;

and when the image shot by the internal camera in real time is displayed, displaying the real-time shot image of the appointed internal camera according to the user instruction.

Preferably, when the position of the food is determined to be changed according to the image shot by the internal camera, the position in the corresponding food AR label is updated.

Preferably, if the food viewed by the user cannot be completely displayed in the images shot by the designated internal camera, the images shot by other internal cameras are selected to display the food viewed by the user.

Preferably, when the external camera photographs foods put in the refrigerator, the method further comprises: the external camera shoots a user, acquires the information of the put-in user, and adds the information of the put-in user into the AR label of the food.

Preferably, the method further comprises: determining that the food is moved out of the refrigerator according to the images shot by the inner camera and the outer camera, determining the position of the food moved out of the refrigerator according to the images shot by the inner camera, and determining the information of the food moved out of the refrigerator according to the position; and updating the AR labels corresponding to the foods moved out of the refrigerator.

Preferably, when the put-in refrigerator food and the out-of-refrigerator food are the same in kind, different in size and/or different in shape, the AR tag of the out-of-refrigerator food is recommended to the user as the AR tag of the put-in refrigerator food.

Preferably, the information of the food comprises shelf life; the method further comprises the following steps: after receiving an inquiry request of a user, screening and displaying food and an AR label thereof according to the shelf life of the food;

and/or the information of the food comprises remark information used for receiving user-defined information input by a user.

Preferably, the shelf life of the food is determined according to user input, or the shelf life of the food is determined according to the putting time and the shelf life of the same food; and/or the presence of a gas in the gas,

when the food is determined to have passed according to the shelf life of the food, the user is prompted that the food has expired.

Preferably, the information of the food includes a putting time, the method further comprising: recommending foods with early putting time in the same type of foods to a user after receiving a recommendation request of the user;

and/or, the method further comprises: and judging the food types at the adjacent positions according to the images shot by the internal camera, and prompting a user to store the food types separately if the food types at the adjacent positions belong to a preset specified combination.

Preferably, the specified combination is vegetables and fruits, and/or the specified combination is uncooked food and cooked food, and/or the specified combination is food and medicine.

Preferably, when the information of the food includes a putting time, the method further includes: and selecting the food in the appointed time stage according to the putting time of the food according to the request of a user, and scratching the selected food out of the background of the shot image according to the image shot by the internal camera for highlighting.

Preferably, the method further comprises: for the food which is determined to be placed in the refrigerator for more than a first set time period according to the placing time of the food and the image shot by the internal camera, prompting a user that the food is possibly overdue; and/or the presence of a gas in the gas,

the method further comprises the following steps: and prompting the user that the food is possibly expired for the food which is determined not to move in the refrigerator beyond a second set time period according to the putting time and the position of the food and the images shot by the internal camera.

Preferably, the method further comprises: and searching price information of the same food on the Internet according to the label information of the food, and prompting a search result to a user.

Preferably, the method further comprises: and selecting food and label information thereof to prompt the user according to the food required to be used in the menu specified by the user.

An apparatus for managing food in a refrigerator, comprising: the device comprises an image identification unit, a setting unit and an interaction unit;

the image identification unit is used for identifying the type and the quantity of food put into the refrigerator according to an image shot by an external camera of the refrigerator; the refrigerator food storage device is also used for determining the position of the food placed in the refrigerator according to the image shot by the internal camera of the refrigerator;

the setting unit is used for generating an AR label aiming at the food put into the refrigerator and recording the information of the food; after receiving a food inquiry request of a user, displaying an AR label of the inquired food; wherein the information of the food at least comprises a food category, a number and a location;

and the interaction unit is used for displaying the image shot by the internal camera in real time after receiving an image viewing request of a user.

According to the scheme, the type and the quantity of the food put into the refrigerator are identified according to the image shot by the external camera of the refrigerator; determining the position of food put into the refrigerator according to an image shot by an internal camera of the refrigerator; generating an AR label for the food put into the refrigerator, and recording the information of the food; after receiving a food inquiry request of a user, displaying an AR label of the inquired food; wherein the information of the food at least comprises a food category, a number and a location. Through the mode, the food information can be more accurately identified for marking through the combination of the inner camera and the outer camera. Further, after receiving an image viewing request of a user, displaying an image shot by the internal camera in real time. Through this mode, can show the real-time image of camera, need not trigger through the switch refrigerator door again and take the latest picture in the refrigerator and show.

Drawings

Fig. 1 is a schematic diagram of a basic flow chart of a method for managing food in a refrigerator according to the present application;

FIG. 2 is a schematic view of a label setting process;

FIG. 3 is a schematic diagram of an information query process;

FIG. 4 is a schematic illustration of an analysis comparison and reminder process;

FIG. 5a, FIG. 5b and FIG. 5c are schematic views of example 1;

FIG. 6 is a schematic view of example 2;

FIG. 7a, FIG. 7b, FIG. 7c and FIG. 7d are schematic diagrams of example 3;

FIG. 8 is a schematic view of example 4;

FIG. 9 is a schematic view of example 5;

FIG. 10 is a schematic view of example 6;

FIG. 11 is a schematic view of example 7;

FIG. 12 is a schematic view of example 8;

FIG. 13 is a schematic view of example 9;

FIG. 14 is a schematic view of example 10;

FIG. 15 is a schematic view of example 11.

Detailed Description

For the purpose of making the objects, technical means and advantages of the present application more apparent, the present application will be described in further detail with reference to the accompanying drawings.

In the present application, a refrigerator is provided with an external camera and an internal camera. This can be combined with the external camera and the internal camera to more accurately recognize food information and user operations. Meanwhile, the number of the internal cameras is at least two, so that images at different angles can be formed.

Fig. 1 is a schematic view of a basic flow of a method for managing food in a refrigerator according to the present application. As shown in fig. 1, the method includes:

step 101, identifying the type and quantity of food put in the refrigerator according to the image shot by the external camera of the refrigerator.

Further, the external camera can further identify the user information of the placer.

Step 102, determining the position of the food put into the refrigerator according to an image shot by an internal camera of the refrigerator;

103, generating an AR label aiming at food put into a refrigerator, and recording the information of the food; after receiving a food inquiry request of a user, displaying an AR label of the inquired food.

Wherein the information of the food at least comprises a food category, a number and a location.

And 104, displaying the image shot by the internal camera in real time after receiving the image viewing request of the user.

The above-mentioned specific process for placing food into the refrigerator may further include the following step 105 when taking food out of the refrigerator.

105, determining that the food is moved out of the refrigerator according to the images shot by the inner camera and the outer camera, determining the position of the food moved out of the refrigerator according to the images shot by the inner camera, and determining a corresponding AR label according to the position; and updating the AR labels corresponding to the foods removed from the refrigerator.

Specifically, the condition for determining that the food is moved out of the refrigerator can be set according to the requirement, for example, the specific operation that the internal camera shoots the hand is performed, and meanwhile, the food is determined to be taken out if the food is found to be invisible according to the image shot by the internal camera.

To implement the management method in the present application described above, the following processing unit may be included in the refrigerator device: the device comprises an image recognition processing unit, an analysis and comparison unit, an interaction unit, a setting unit, a storage unit, an inquiry unit and a reminding unit.

The following is a detailed description of the functions and processes of the respective units.

1. Image recognition processing unit

This unit mainly used people discernment, refrigerator food discernment and AR panorama generation. The method mainly comprises the following steps:

1) the refrigerator is provided with a plurality of cameras, the positions of food are identified through an inner camera, and the types, the quantity and the users of the food are identified through an outer camera;

2) the image recognition can record the current image of an object and can also carry out real-time positioning on the position of the object in a three-dimensional space through a plurality of cameras;

3) after image recognition, the main body is separated from the background, and highlighted after image matting is used for displaying food required by a user or being searched.

2. Analysis and comparison unit

The unit is used for recording, analyzing and comparing the change of the AR image in the refrigerator. The method mainly comprises the following steps:

1) analyzing and comparing the position change of food in the refrigerator in real time, and automatically updating the panoramic image;

2) the label is cancelled for the non-existing article, the position of the moving article is tracked, and the same article is taken out, changed in shape or size and put in again (such as a half-cut watermelon) to be regarded as a new article, and the label needs to be reset. After the system can compare and judge through an image recognition technology, if the categories of the food put in and the food removed out are the same, label information (such as food names and the like) of the food removed out can be recommended to a user, and the information can be rapidly input.

That is, when the kind, size and/or shape of the refrigerator food put in and the refrigerator food removed are the same, the AR tag of the refrigerator food removed is recommended to the user as the AR tag of the refrigerator food put in.

3) When food is put in, putting time can be included in the food information, the putting time of the same food is compared and identified, and the food put in first is recommended to be eaten by a user;

4) and identifying the type of the adjacent food, and reminding the user to store the food separately if the adjacent food is detected to be the set food combination. For example, vegetables and fruits (fruits release ethylene and vegetables turn yellow), raw and cooked foods can remind users to store separately, and foods and medicines can also be recommended to store separately;

5) whether the food product is likely to expire is determined by identifying items that have not disappeared or moved for a long period of time. After a period of time, the areas where the articles move most frequently and the areas where the articles move less can be counted for marking. In more detail, for the food which is determined to be placed in the refrigerator for a time exceeding a first set time period according to the placing time of the food and the image shot by the internal camera, prompting a user that the food may be out of date; and/or, for the food which is determined not to move in the refrigerator beyond a second set time period according to the putting time and the position of the food and the image shot by the internal camera, prompting the user that the food is possibly expired.

3. Interaction unit

The unit is mainly used for identifying touch/non-touch operation performed by a user in the process of operating the panoramic image. The main interactions are as follows:

1) the touch operation comprises touch screen operations such as clicking and sliding, and is used for checking, screening, information input and the like of articles;

2) the non-touch operation comprises actions such as checking and selecting the panoramic image in an air space, and is realized based on a three-dimensional gesture recognition technology;

this unit provides an automatic AR feedback effect:

1) for occluded objects, the system will automatically "switch viewing angles" for the user. Rotating the direction of the panoramic picture and adjusting the visual angle, so that a user can view at an angle favorable for observation; in more detail, if the food viewed by the user cannot be displayed completely in the photographed image of the designated internal camera, images photographed by other internal cameras are selected to display the food viewed by the user.

2) For foods that have expired, there is a ring of AR special effects around the area or item, alerting the user that treatment is needed here.

3) Aiming at the sterilization scene, the user can see the sterilization dynamic effect generated by the surrounding layer of food, and the sterilization process and the sterilization state can be viewed from different positions by adjusting the angle.

4. Setting unit

The unit is mainly used for setting and recording food labels. The method mainly comprises the following steps:

1) AR label sets up the module: generating an AR label of the food, and binding the label with the article; when the position of the object is changed, the position of the label is changed; after the article is taken away, the original label automatically disappears.

The setting object includes:

-label settings of individual articles: an external camera on a refrigerator door identifies an article, and the article is placed in the refrigerator and then automatically generates a label and is bound;

-label settings of a plurality of different items: an external camera on a refrigerator door identifies the types and the quantity of articles (one apple and one pear), and the labels are sequentially and automatically generated and bound after the articles are placed in the refrigerator;

-label settings of a plurality of identical items: an external camera on a refrigerator door identifies the types and the quantity of articles, and the articles are automatically grouped after being placed in the refrigerator to generate and bind labels; if the automatic grouping fails, a user can manually draw a group range and pack a plurality of same articles in a limited area into a whole for setting;

-if it is a pack of items (a pack of fruit) or food contained by a vessel, by default identifying as a whole and creating a label;

2) an automatic recording module: automatically inputting the date of placing the article, user information, article type, image, quantity and the like; automatically judging whether the article which is taken out and changed in shape or size and put in is the same as the article before or not, and if the article is the recommended name, giving the user;

3) the manual recording module: manually entering information, such as detailed shelf life and the like;

5. memory cell

This unit mainly used storage information mainly includes:

1) an image storage module: AR panorama information for storing refrigerator foods;

2) a user information storage module: for storing user information, including face and height information;

3) a tag information storage module: the AR label information used for storing setting and updating;

4) the menu storage module: the system is used for storing menu information which can be formed by various raw materials;

5) food collocation storage module: for storing which foods may not be put in the refrigerator or put together;

6) the interactive instruction storage module: the touch control device is used for storing corresponding gestures and touch instructions operated by a user;

6. query unit

The unit is mainly used for collecting the interactive instructions, matching the interactive mode identified by the interactive unit with the preset interactive mode of the storage unit, and triggering the corresponding instructions to perform related operations if the interactive modes are the same. The method mainly comprises the following steps:

1) an image viewing module: collecting interaction instructions and carrying out panorama viewing related operations; the occluded object is presented to the user in a mode of 'converting the viewing angle', and the user can view the object at the other side;

2) a shelf life inquiry module: collecting interactive instructions and screening and displaying food according to shelf life; if the user does not input the shelf life information, the system inquires the date automatically recorded on the day when the article is put in, compares the possible expiration times of different foods and judges the fresh state of the foods;

3) AR label inquiry module: collecting an interactive instruction, executing query operation, and displaying detailed label information;

4) the menu raw material query module: collecting the interactive instructions, executing query operation, and highlighting the information of the required raw materials;

7. a reminding unit:

this unit is mainly used for realizing the warning function of food state. The method mainly comprises the following steps:

1) and (4) overdue reminding: reminding about the food which is about to expire in a visual label mode; in combination with the time of putting the food in, whether the food is possibly overdue can be judged by identifying articles which are not disappeared or moved for a long time, and the expiration time can be calculated by manually entering the shelf life and giving a reminder. For foods that have expired, there is a ring of AR special effects around the area or item, alerting the user that treatment is needed here.

2) And (4) preferential edible recommendation: identifying the time of putting the same food in the two times, and recommending the food put in the first time to a user for eating;

3) placing and reminding: after the food types are identified, giving a prompt for foods which are not suitable to be put together;

4) price reminding: the label can automatically display the commodity price information of a nearby supermarket and give a prompt when a discount exists (for example, egg discount, label pop-up discount information)

The above is the function and processing of each unit.

The management method of the present application can be divided into a plurality of flows according to different requirements to complete corresponding processing, which is described in detail below.

1. Information identification and storage process:

1) the image recognition processing unit recognizes user information including face and height information;

2) the image recognition processing unit acquires information such as food images, types, quantity and the like;

3) when a user puts in an article, the setting unit automatically generates an AR label and inputs time information when the article is put in, and other information is manually recorded;

4) the image recognition processing unit carries out three-dimensional space position positioning on the food; the analysis and comparison unit analyzes images and the spatial position changes of the images when the articles are put in and taken out; the image identification processing unit generates and updates an AR panorama according to the image information;

5) user information, panoramic images, food location information, and AR tag information are stored in the storage unit.

2. Label setup procedure (as shown in fig. 2):

identifying food

An external camera on the refrigerator door identifies the type and the number of the articles;

putting into food

1) Individual article: when a user puts in a single food, the system automatically generates and binds a label, the image recognition processing unit generates an AR panorama, and the information is stored in the storage unit after being confirmed;

2) a plurality of different articles: a user puts a plurality of different varieties of foods, and the system automatically generates and binds labels in sequence after putting the foods; the image recognition processing unit generates an AR panorama, and the information is stored in the storage unit after being confirmed;

3) a plurality of identical items: a user puts a plurality of foods of the same variety into a refrigerator and then automatically groups the foods to generate labels and bind the labels; if the automatic grouping fails, a user can manually draw a group range and pack a plurality of same articles in a limited area into a whole for setting;

4) if the food is a pack of articles (a pack of fruits) or food contained in a vessel, the food is identified as a whole by default and a label is established;

mobile food

1) The user moves food, the analysis and comparison module compares the position change of the articles in the refrigerator, and the label is found out;

2) a label setting module of the setting unit updates label information, cancels labels for non-existing articles, and tracks the positions of moving articles; if the same article is taken out and changed in shape or size and then put in again (such as a half-cut watermelon), the article is regarded as a new object and the label needs to be reset. The system can recommend the possible names of the user objects after comparison and judgment through an image recognition technology, and help to quickly input information; after the information is confirmed, storing the information in a storage unit;

3. information query flow (as shown in fig. 3):

1) the interaction unit identifies a user gesture command and divides the user gesture command into a touch command and a non-touch command;

2) the query unit checks the AR panoramic image according to the user instruction, screens and displays food through the quality guarantee period, displays detailed label information,

highlighting the menu raw materials according to the menu selection instruction of the interaction unit;

4. analysis comparison and reminder flow (as shown in fig. 4):

1) the interaction unit identifies the user action, the analysis and comparison unit compares and identifies the time of putting the same food twice before and after, and the reminding unit recommends the food put in first to be eaten by the user;

2) the interaction unit identifies the action of a user, the analysis and comparison unit identifies the types of adjacent foods in the refrigerator, if vegetables and fruits are detected to be put together, uncooked foods and cooked foods are put together, the reminding unit reminds the user to store the foods and the medicines separately (the fruits release ethylene and the vegetables turn yellow), and the foods and the medicines are also recommended to store separately;

3) the system finds out foods (fresh, used as soon as possible and not fresh) at different time stages according to the instruction of the interaction unit, the image recognition processing unit extracts the picture outline from the background, and the reminding unit gives a reminding in a highlight display mode;

4) the interaction unit identifies user actions, identifies items that are long-lived or not moved to determine that the food may be out of date. After a period of time, the areas where the articles rarely move are counted and marked to remind the user.

Specific examples of several procedures are given below.

Example 1: panorama interaction (as shown in fig. 5 a-5 c):

step 1: a picture is shot by a camera to form a 360-degree omnibearing panorama (figure 5a), and a user can select any visual angle to view according to needs, wherein the view comprises two side doors and the back;

step 2: viewing the food item within the panorama through gesture interaction and touch interaction (fig. 5 b);

and step 3: and automatically carrying out view angle conversion, adjusting the viewing position and angle of the user of the target object and displaying the AR label content (figure 5 c).

Example 2: panorama update (as shown in fig. 6):

step 1: judging the position of an object through a plurality of cameras in the refrigerator door;

step 2: judging what is moved according to the comparison between the position of the user when putting down the article and the position information after moving, and updating the image in real time;

and step 3: if the children are taken away, the external camera judges who takes away the articles when the refrigerator door is closed, and if the children can inform the parents of the information;

example 3: AR tag settings (as shown in fig. 7 a-7 d):

step 1: the external cameras on the refrigerator door identify the type and number of the articles (fig. 7 a);

step 2: put into food (fig. 7b)

1) Individual article: when a user puts in a single food, the system automatically generates a label and binds the label;

2) a plurality of different articles: a user puts a plurality of different varieties of foods, and the system automatically generates and binds labels in sequence after putting the foods;

and step 3: mobile food (fig. 7c)

1) The user moves food, AR label information is updated in real time according to article changes, labels are cancelled for non-existing articles, position tracking is carried out on the moving articles, and when the positions of the articles change, the positions of the labels change;

2) if the same article is taken out and changed in shape or size and then put in again (such as a half-cut watermelon), the article is regarded as a new object and the label needs to be reset. The system can recommend the possible names of the user objects after comparison and judgment through an image recognition technology, and help to quickly input information; after the information is confirmed, storing the information in a storage unit;

and 4, step 4: the label status will change with the freshness of the item, e.g. green fresh, yellow not fresh, red expired (fig. 7 d).

Example 4: food lookup by food list/voice (as shown in fig. 8):

step 1: the user searches for food through list selection or voice;

step 2: the system finds a target and extracts an object from the background, and highlights are displayed;

and step 3: automatically converting the visual angle, adjusting the user viewing position and angle of the target object and displaying the label content;

figure 4 food lookup: by food listing/voice lookup.

Example 5: find food by recipe (as shown in fig. 9):

step 1: the user selects a menu;

step 2: the system inquires and highlights the food needed in the recommended menu in the refrigerator;

and step 3: if the material is missing, the purchase of the missing recipe food in the refrigerator is prompted.

Example 6: food screening (as shown in figure 10):

step 1: the user screens and searches through shelf life classification icons (fresh, not fresh, expired);

step 2: the system highlights all food items that meet the criteria.

Example 7: food was recommended according to the order of consumption (as shown in fig. 11):

step 1: identifying the two-time putting time before and after the same food;

step 2: recommending the food put in first to be eaten by the user;

example 8: reminding: shelf life determination and reminder (as shown in fig. 12):

step 1: if the user does not enter shelf life information, the refrigerator determines that the food may be out of date by identifying items that have not moved or disappeared for a long period of time.

Step 2: after a period of time, the regions that move very little are marked. For foods that have expired, there is a ring of AR special effects around the area or item, alerting the user that treatment is needed here.

Example 9: reminding: food placement reminder (as shown in fig. 13):

step 1: a user puts food in;

step 2: identifying and analyzing food types by images;

and step 3: if it is detected that the vegetables and fruits are put together and the uncooked food and the cooked food are put together, an AR effect is provided to remind the user to store them separately.

Example 10 reminder: price change reminder (as shown in fig. 14):

step 1: when discount preferential information exists, the food label is automatically popped up and converted to a visual angle suitable for being checked by a user;

step 2: clicking the label to display the position and commodity price information of the nearby supermarket

Example 11: visual AR sterilization (as shown in fig. 15):

step 1: selecting sterilization by a user;

step 2: the user views the dynamic effect of sterilization generated around each layer of food;

and step 3: the user adjusts the angle to view the sterilization state, progress and detailed information from different positions.

The foregoing is a specific implementation of the management method in the present application. In view of the foregoing, the present application discloses a method for managing refrigerator foods based on an AR panorama. Based on panorama interaction, realize the convenient management to storing food in the refrigerator.

Specifically, the method forms an AR panorama of the articles in the refrigerator through multi-camera identification, sets an AR information label for the food put in the refrigerator, binds the label with the articles and the space positions of the articles, compares and monitors the positions of the articles in real time, and updates the states of the articles. The management content comprises the following steps: setting labels, searching foods, screening foods, recommending eating, dynamically sterilizing and reminding. The interactive feedback is presented in the form of an AR effect. By the mode, the food information is visually managed, and a user can more conveniently know the information stored in the food in the refrigerator.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A method for managing food in a refrigerator, comprising:

determining the positions of the foods put into the refrigerator according to images shot by a plurality of internal cameras positioned at different angles and positions of the refrigerator; the position of the food placed in the refrigerator is a three-dimensional space position determined according to images shot by all internal cameras;

generating an AR label aiming at the food put into the refrigerator, recording the information of the food, and generating an AR panorama according to images shot by all internal cameras; after receiving a food inquiry request of a user, displaying an AR label of the inquired food; wherein the information of the food at least comprises a food category, a number and a location;

after receiving an image viewing request of a user, displaying an image shot by the internal camera in real time;

the method further comprises the following steps: and determining that the food is moved out of the refrigerator according to the images shot by the inner camera and the outer camera.

2. The method according to claim 1, wherein the live view image of the specified internal camera is displayed according to a user instruction while the live view image of the internal camera is displayed.

3. The method of claim 1, wherein the position in the corresponding food AR tag is updated when it is determined from the image taken by the internal camera that the position of the food has changed.

4. The method of claim 2, wherein if the food viewed by the user cannot be displayed completely in the images captured by the designated internal camera, selecting the images captured by the other internal cameras to display the food viewed by the user.

5. The method of claim 1, wherein when the external camera photographs food put in the refrigerator, the method further comprises: the external camera shoots a user, acquires the information of the put-in user, and adds the information of the put-in user into the AR label of the food.

6. The method of claim 1, further comprising: determining the position of the refrigerator food to be moved out according to the image shot by the internal camera, and determining the information of the refrigerator food to be moved out according to the position; and updating the AR labels corresponding to the foods moved out of the refrigerator.

7. The method of claim 6, wherein the AR tag of the removed refrigerator food is recommended to the user as the AR tag of the inserted refrigerator food when the inserted refrigerator food is the same kind, size and/or shape as the removed refrigerator food.

8. The method of claim 1, wherein the information of the food comprises a shelf life; the method further comprises the following steps: after receiving an inquiry request of a user, screening and displaying food and an AR label thereof according to the shelf life of the food;

9. The method of claim 8, wherein the shelf life of the food is determined based on user input, or wherein the shelf life of the food is determined based on the time of insertion and the shelf life of the same type of food; and/or the presence of a gas in the gas,

10. The method of claim 1, wherein the information about the food includes a placement time, the method further comprising: recommending foods with early putting time in the same type of foods to a user after receiving a recommendation request of the user;

11. The method according to claim 10, wherein the specified combinations are vegetables and fruits, and/or wherein the specified combinations are uncooked and cooked foods, and/or wherein the specified combinations are food products and pharmaceutical products.

12. The method of claim 10, wherein when the information about the food includes a placement time, the method further comprises: and selecting the food in the appointed time stage according to the putting time of the food according to the request of a user, and scratching the selected food out of the background of the shot image according to the image shot by the internal camera for highlighting.

13. The method of claim 1, further comprising: for the food which is determined to be placed in the refrigerator for more than a first set time period according to the placing time of the food and the image shot by the internal camera, prompting a user that the food is possibly overdue; and/or the presence of a gas in the gas,

14. The method of claim 1, further comprising: and searching price information of the same food on the Internet according to the label information of the food, and prompting a search result to a user.

15. The method of claim 1, further comprising: and selecting food and label information thereof to prompt the user according to the food required to be used in the menu specified by the user.

16. An apparatus for managing food in a refrigerator, comprising: the device comprises an image identification unit, a setting unit and an interaction unit;

the image identification unit is used for identifying the type and the quantity of food put into the refrigerator according to an image shot by an external camera of the refrigerator; the refrigerator food storage device is also used for determining the positions of the food placed in the refrigerator according to images shot by a plurality of internal cameras positioned at different angles and positions of the refrigerator; the position of the food placed in the refrigerator is a three-dimensional space position determined according to images shot by all internal cameras; the refrigerator is also used for determining that the food is moved out of the refrigerator according to the images shot by the inner camera and the outer camera;

the setting unit is used for generating an AR label aiming at the food put into the refrigerator, recording the information of the food and generating an AR panorama according to images shot by all internal cameras; after receiving a food inquiry request of a user, displaying an AR label of the inquired food; wherein the information of the food at least comprises a food category, a number and a location;