CN111292828A

CN111292828A - Intelligent refrigerator and food material management method, device and storage medium thereof

Info

Publication number: CN111292828A
Application number: CN201811494588.5A
Authority: CN
Inventors: 姬四林; 彭红亮; 刘兆祥; 陈红欣
Original assignee: Hisense Shandong Refrigerator Co Ltd
Current assignee: Hisense Shandong Refrigerator Co Ltd
Priority date: 2018-12-07
Filing date: 2018-12-07
Publication date: 2020-06-16

Abstract

The invention discloses an intelligent refrigerator and a food material management method, a food material management device and a storage medium thereof, wherein the intelligent refrigerator comprises: the door body is used for opening and closing the storage box; the refrigerator food storage device further comprises an image acquisition device, a display device and a control unit, wherein the image acquisition device is used for shooting and acquiring images containing food materials in the refrigerator; the control unit identifies and compares a first image acquired after the door body is closed with a second image acquired last time, obtains food materials added or taken out when the door body is opened, displays food material icons of the food materials added at this time at corresponding positions of the virtual storage display area, and deletes the food material icons of the food materials taken out at this time in the virtual storage display area. Identifying and comparing the first image acquired this time with the second image acquired last time, and determining the food materials added or taken out when the door body is opened this time; and the food material icons corresponding to the food materials are added or deleted in the virtual storage display area, so that the management of the food materials is completed, and the efficiency of food material management is improved.

Description

Intelligent refrigerator and food material management method, device and storage medium thereof

Technical Field

The invention relates to the technical field of image automatic identification technology and household appliances, in particular to an intelligent refrigerator, a food material management method and device thereof and a storage medium.

Background

With the gradual development of science and technology, intelligent refrigerators attract more and more attention of people. An intelligent refrigerator generally includes a voice acquisition module, a Wi-Fi module, a camera, a touch display screen, and the like. The user can carry out operations such as video playing, food material management and weather playing aiming at the intelligent refrigerator.

When a target food material is managed in the prior art, a user firstly holds a food material to be put into a refrigerator by hand and shoots the food material by aiming at a camera on the refrigerator to form a food material image, and the food material image is displayed on a display screen of the refrigerator. In addition, when the icon is added, the user needs to add the icon of the target food material from the food material database to the virtual storage grid display interface of the refrigerator in a manual dragging mode, so that food material management is realized.

However, the technical defect of the scheme is that a user holds the food materials by hand to take a picture, and needs to manually drag an icon of the target food material from the food material database to a food management interface, and if the number of the food materials is large, the user needs to drag the icon for multiple times, so that the efficiency of food material management is reduced.

Disclosure of Invention

The embodiment of the invention provides an intelligent refrigerator and a food material management method, device and storage medium thereof, which are used for solving the problem that in the prior art, the efficiency of food material management is reduced because icons of target food materials need to be manually photographed and dragged to a virtual storage grid display interface of the refrigerator.

The embodiment of the invention provides an intelligent refrigerator, which comprises: the storage box is arranged on the door body; the device also comprises an image acquisition device, a display device and a control unit;

the image acquisition device is used for shooting and acquiring an image containing food materials in the refrigerator; the control unit controls the image acquisition device to acquire a first image containing food materials in the refrigerator after the detected refrigerator door body is closed; the display device is positioned on the outer side of the refrigerator door body and used for displaying a food material management interface; the food material management interface comprises a food material real-time shooting display area and a virtual storage display area, wherein the food material real-time shooting display area is used for displaying images shot by the image acquisition device; the virtual storage display area is used for displaying food material icons of food materials stored in the refrigerator;

the control unit is further configured to identify and compare a first image acquired after the door body is closed with a second image acquired last time, obtain food materials added or taken out when the door body is opened, display food material icons of the food materials added at this time at corresponding positions of the virtual storage display area, and delete the food material icons of the food materials taken out at this time in the virtual storage display area.

Further, the control unit is further configured to determine the food material selected by the user and acquire food material information of the food material when receiving a selection operation of the user on the real-time food material shooting display area;

the display device is also used for displaying the food material information.

Further, the control unit is further configured to highlight the food material icon corresponding to the food material selected by the user in the virtual storage display area when receiving a selection operation of the user on the food material real-time shooting display area.

Further, the food material information comprises at least one of:

nutrition information, shelf life information, recipe information.

Further, the display device further comprises a food material database display area, and the food material database display area is used for displaying icons of various food materials.

Further, the control unit is specifically configured to identify each food material image in the first image and the second image, form a plurality of first food material image identification frames in the first image, and form a plurality of second food material image identification frames in the second image; comparing any one first food material image recognition frame with any one second food material image recognition frame, and if the similarity between the first food material image recognition frame and any one second food material image recognition frame is smaller than a set threshold value, determining that the food materials in the first food material image recognition frame are the food materials put in the door body when the door body is opened; comparing any one of the second food material image recognition frames with the first food material image recognition frame, and if the similarity between the second food material image recognition frame and any one of the first food material image recognition frames is smaller than a set threshold value, the food material in the second food material image recognition frame is the food material taken out by opening the door body.

Further, the control unit is further configured to, for each region in the first image, determine whether the article identifiers of the region and the food materials corresponding to candidate regions adjacent to the region are the same according to the article identifier corresponding to each region; if the distance between the area and the candidate area is the same as the preset distance threshold, judging whether the distance between the area and the candidate area is smaller than the preset distance threshold; if so, merging the candidate region into the region, and taking the merged region as a food material image recognition frame corresponding to the food material of the item identifier.

Further, the control unit is specifically configured to determine information of a target shelf of the food material in the refrigerator according to the position information of the food material in the first image for the food material added when the door body is opened at this time; and displaying the food material icon added this time in a display area of the virtual storage display area corresponding to the target shelf.

Further, the control unit is further configured to determine a duration for which the food material corresponding to each food material icon is managed according to the stored first initial time and current time for which each managed food material is managed; and displaying corresponding duration at the corresponding position of each food material icon in the virtual storage display area.

Further, the control unit is further configured to determine, for the determined duration corresponding to each food material icon, a duration management range matched with the duration according to each preset duration management range, and display the duration at a position corresponding to the food material icon in the virtual storage display area by using a font color corresponding to the duration management range.

The embodiment of the invention provides a food material management method of an intelligent refrigerator, which comprises the following steps:

after the detected refrigerator door body is closed, controlling an image acquisition device to acquire a first image containing food materials in the refrigerator, wherein the image acquisition device is used for shooting and acquiring the image containing the food materials in the refrigerator;

identifying and comparing the first image obtained this time with the second image obtained last time, obtaining the food materials added or taken out when the door body is opened, displaying the food material icons of the food materials added this time at the corresponding positions of the virtual storage display area of the refrigerator, and deleting the food material icons of the food materials taken out this time in the virtual storage display area.

Further, the method further comprises:

and when receiving a selection operation of a user on the food real-time shooting display area, determining the food material selected by the user, and acquiring and displaying food material information of the food material.

Further, the method further comprises:

and highlighting the food material icon corresponding to the food material selected by the user in the virtual storage display area.

Further, the food material information comprises at least one of:

nutrition information, shelf life information, recipe information.

Further, identifying and comparing the first image obtained this time with the second image obtained last time, and obtaining the food material added or taken out when the door body is opened includes:

identifying each food material image in the first image and the second image, forming a plurality of first food material image identification frames in the first image, forming a plurality of second food material image identification frames in the second image, comparing any one first food material image identification frame with any one second food material image identification frame, and if the similarity between the first food material image identification frame and any one second food material image identification frame is smaller than a set threshold value, determining the food materials in the first food material image identification frame as the food materials put in the door body when the door body is opened; comparing any one of the second food material image recognition frames with the first food material image recognition frame, and if the similarity between the second food material image recognition frame and any one of the first food material image recognition frames is smaller than a set threshold value, the food material in the second food material image recognition frame is the food material taken out by opening the door body.

Further, the forming of the plurality of food material image recognition frames in the image includes:

aiming at each area in the image, judging whether the area is the same as the item identifier of the food material corresponding to the candidate area adjacent to the area or not according to the item identifier corresponding to each area; if the distance between the area and the candidate area is the same as the preset distance threshold, judging whether the distance between the area and the candidate area is smaller than the preset distance threshold; if so, merging the candidate region into the region, and taking the merged region as a food material image recognition frame corresponding to the food material of the item identifier.

Further, the displaying food material icons of the food materials added this time at corresponding positions of the virtual storage display area of the refrigerator includes:

for the food materials added when the door body is opened at this time, determining the information of a target shelf of the food materials in the refrigerator according to the position information of the food materials in the first image; and displaying the food material icon added this time in a display area of the virtual storage display area corresponding to the target shelf.

Further, the method further comprises:

determining the time length of the managed food material corresponding to each food material icon according to the stored first initial time and current time of each managed food material; and displaying corresponding duration at the corresponding position of each food material icon in the virtual storage display area.

Further, the method further comprises:

and aiming at the determined time length corresponding to each food material icon, determining a time length management range matched with the time length according to each preset time length management range, and displaying the time length at the position corresponding to the food material icon in the virtual storage display area by adopting the font color corresponding to the time length management range.

The embodiment of the invention provides a food material management device of an intelligent refrigerator, which comprises:

the control module is used for controlling the image acquisition device to acquire a first image containing food materials in the refrigerator after the detected refrigerator door body is closed, wherein the image acquisition device is used for shooting and acquiring the image containing the food materials in the refrigerator;

the management module is used for identifying and comparing the first image acquired this time with the second image acquired last time to acquire the food materials added or taken out when the door body is opened, displaying food material icons of the food materials added this time at corresponding positions of the virtual storage display area of the refrigerator, and deleting the food material icons of the food materials taken out this time in the virtual storage display area.

The invention provides an intelligent refrigerator, which comprises: the system comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

the memory has stored therein a computer program which, when executed by the processor, causes the processor to perform the steps of any of the methods described above.

The invention provides a computer readable storage medium storing a computer program executable by an intelligent refrigerator, the program, when run on the intelligent refrigerator, causing the intelligent refrigerator to perform the steps of any of the methods described above.

The embodiment of the invention provides an intelligent refrigerator and a food material management method, device and storage medium thereof, wherein after a refrigerator door is detected to be closed, a first image containing food materials in the refrigerator is obtained by controlling an image acquisition device, wherein the image acquisition device is used for shooting and obtaining the image containing the food materials in the refrigerator; identifying and comparing the first image obtained this time with the second image obtained last time, obtaining the food materials added or taken out when the door body is opened, displaying the food material icons of the food materials added this time at the corresponding positions of the virtual storage display area of the refrigerator, and deleting the food material icons of the food materials taken out this time in the virtual storage display area. In the embodiment of the invention, identification comparison is carried out on the basis of the first image obtained at this time and the second image obtained last time, and the food materials added or taken out at this time are determined; and the food material icons corresponding to the food materials are added or deleted in the virtual storage display area, so that the management of the food materials is completed, and the efficiency of food material management is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic structural diagram of an intelligent refrigerator provided in embodiment 1 of the present invention;

fig. 2 is a schematic structural diagram of the outside of an intelligent refrigerator provided in embodiment 1 of the present invention;

fig. 3 is a schematic flow chart illustrating a food material management method for an intelligent refrigerator according to embodiment 2 of the present invention;

fig. 4 is a schematic view illustrating a user clicking food materials and acquiring food material information according to embodiment 3 of the present invention;

fig. 5 is a schematic view of a menu interface of a display screen of a refrigerator according to embodiment 3 of the present invention;

fig. 6 is one of schematic diagrams of a display screen of a refrigerator provided in embodiment 6 of the present invention;

fig. 7 is a second schematic view of a display screen of a refrigerator according to embodiment 6 of the present invention;

fig. 8 is a flowchart illustrating a food material management method for an intelligent refrigerator according to embodiment 6 of the present invention;

fig. 9 is a schematic structural view of a food material management device of an intelligent refrigerator according to embodiment 7 of the present invention;

fig. 10 is a schematic structural diagram of an intelligent refrigerator provided in embodiment 8 of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the attached drawings, and it should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example 1:

the embodiment of the invention provides an intelligent refrigerator, which is shown in fig. 1 and fig. 2, and comprises a storage box 1 and a door body 2 for opening and closing the storage box 1, wherein the storage box 1 is used for storing various food materials, and the intelligent refrigerator further comprises: the device comprises an image acquisition device 3, a display device 4 and a control unit;

specifically, the image acquisition device 3 is used for shooting and acquiring an image containing food materials in the refrigerator;

after the detected closing of the refrigerator door body 2, the control unit controls the image acquisition device 3 to acquire a first image containing food materials in the refrigerator;

the display device 4 is positioned on the outer side of the refrigerator door body 2 and used for displaying a food material management interface; the food material management interface comprises a food material real-time shooting display area and a virtual storage display area, wherein the food material real-time shooting display area is used for displaying images shot by the image acquisition device 3; the virtual storage display area is used for displaying food material icons of food materials stored in the refrigerator;

the control unit is further configured to identify and compare a first image acquired after the door body 2 is closed with a second image acquired last time, obtain food materials added or taken out when the door body 2 is opened, display food material icons of the food materials added at this time at corresponding positions of the virtual storage display area, and delete the food material icons of the food materials taken out at this time in the virtual storage display area.

Specifically, the intelligent refrigerator provided by the embodiment of the invention comprises a storage box 1 and a door body 2 for opening and closing the storage box 1, wherein the storage box 1 is used for storing various food materials; in addition, the intelligent refrigerator is also provided with an image acquisition device 3, a display device 4 positioned on the outer side of the refrigerator door body 2 and a control unit. The image acquisition device 3 is used for acquiring images of food materials in the refrigerator; the control unit is used for controlling the image acquisition device 3 and obtaining food materials added or taken out by a user; the display device 4 further includes a virtual storage display area and a food material database display area. The image displayed in the food real-time shooting display area is a first image obtained after the door body 2 is closed, and food material icons corresponding to each food material in the first image are displayed in the virtual storage display area.

Specifically, this image acquisition device 3 can set up in this refrigerator door 2 inboard, perhaps image acquisition device 3 also can set up inside storage box 1, and the preferred, as shown in fig. 1 this image acquisition device 3 sets up in refrigerator door 2 inboard, and this image acquisition device 3 sets up the camera that has the shooting function, and the camera can be shot to the edible material in the refrigerator. In order to prevent the conditions that the food materials are not completely photographed or are shielded, a plurality of cameras can be arranged in the image acquisition device 3, and the number of the adopted cameras can be determined according to the size of the space needing to be managed in the refrigerator and the photographing range of the cameras. If the inner side of the refrigerator door body 2 is provided with a plurality of cameras, image splicing can be carried out according to the images collected by each camera to obtain a first image. The process of image stitching of images acquired by a plurality of cameras is the prior art, and details of the process are not repeated in the embodiment of the present invention.

Preferably, as shown in fig. 1, in the camera structure in the image acquisition device 3 inside the refrigerator door 2, since three layers of shelves are configured in the refrigerating chamber of a general refrigerator for storing food materials, in order to more clearly photograph the food materials in the refrigerating chamber of the refrigerator, 3 cameras can be arranged on the image acquisition device 3 inside the refrigerator door 2, and the 3 cameras are arranged on the inside of the door 2 in the refrigerating chamber of the refrigerator at equal intervals. The method comprises the steps that a first camera collects a food material image 1, a second camera collects a food material image 2, a third camera collects a food material image 3, and the obtained food material image 1, the food material image 2 and the food material image 3 are spliced by using an image splicing technology to obtain a first image.

Example 2:

fig. 3 is a schematic flow chart of a food material management method for an intelligent refrigerator according to an embodiment of the present invention, where the process includes the following steps:

s301: after the detected refrigerator door body 2 is closed, controlling an image acquisition device 3 to acquire a first image containing food materials in the refrigerator, wherein the image acquisition device 3 is used for shooting and acquiring the image containing the food materials in the refrigerator.

As shown in fig. 2, the food material management method is applicable to an intelligent refrigerator with a display screen 4, specifically, an intelligent refrigerator capable of managing food materials put in the refrigerator, for example, an intelligent refrigerator capable of managing food materials in the refrigerator such as expiration reminding.

Generally, when a user opens the refrigerator, food materials are placed in or taken from the refrigerator, and therefore, in order to manage the food materials, if the control unit recognizes that the refrigerator door 2 is closed, it is necessary to determine whether the unmanaged food materials exist. Specifically, the control unit in the intelligent refrigerator has the function of sensing the state that the refrigerator door body 2 opened or closed, and after the control unit identified the refrigerator door body 2 and closed, control image acquisition device 3 in the intelligent refrigerator and acquire the first image that contains the edible material in the refrigerator, specifically, through the camera that has the shooting function in image acquisition device 3, this image acquisition device 3 is used for shooting and acquires the image that contains the edible material in the refrigerator.

And after the control unit acquires the first image, displaying the first image in a food material real-time shooting display area in a food material management interface.

S302: identifying and comparing the first image obtained this time with the second image obtained last time, obtaining the food materials added or taken out when the door body 2 is opened, displaying the food material icons of the food materials added this time at the corresponding positions of the virtual storage display area of the refrigerator, and deleting the food material icons of the food materials taken out this time in the virtual storage display area.

In order to facilitate the control unit to determine the food materials added and/or taken out each time the door is opened, a second image which is acquired after the door body 2 of the refrigerator is closed last time and is subjected to image recognition is stored in the control unit, and after the first image is obtained, the first image is subjected to image recognition operation to determine each food material contained in the first image. Therefore, the food materials added when the door body 2 is opened are determined according to the food materials added in the first image relative to the second image, and the food materials taken out when the door body 2 is opened are determined according to the food materials reduced in the first image relative to the second image. And searching the food material icons of the added or taken out food materials from the food material database in the food material database display area, displaying the food material icons of the food materials added this time in the virtual storage display area of the refrigerator, or deleting the food material icons of the food materials taken out this time in the virtual storage display area of the refrigerator, so that the management of the food materials in the refrigerator is realized, and the experience effect of a user is improved. After the food material icons corresponding to the added food materials or the taken-out food materials are displayed or deleted, the refrigerator can automatically store the corresponding relations of the food materials and the food material icons corresponding to the food materials in a virtual storage display area of the refrigerator.

Specifically, in a possible implementation manner, a communication module may be disposed in the control unit of the intelligent refrigerator, and the communication module is used for the intelligent refrigerator and the server to transmit information. After the first image is obtained, the intelligent refrigerator uploads the first image to a server with an image recognition function, the server performs image recognition operation on the first image, for example, a cloud server, after the server obtains the item identifiers corresponding to all food materials in the first image, the server sends the item identifier information and information of the area corresponding to the item identifier information in the first image to the intelligent refrigerator together, and a control unit of the intelligent refrigerator receives the item identifier information sent by the server and the information of the area corresponding to the item identifier information in the first image. After receiving and storing the article identification information, the control unit of the intelligent refrigerator displays the first image on a food material real-time shooting display area of the refrigerator. The control unit determines that the refrigerator door body 2 closes the food materials added or taken out by the user at this time by comparing the recognition results of the first image and the second image, displays the food material icons of the added food materials in the virtual storage display area of the refrigerator, and deletes the food material icons of the taken-out food materials in the virtual storage display area of the refrigerator.

In another possible embodiment, an image recognition module may be disposed in the control unit of the intelligent refrigerator, and the image recognition module may perform an image recognition operation on the first image captured by the image capturing device 3. After the control unit acquires the first image, the control unit performs recognition operation on the first image through the image recognition module, recognizes the article identifiers corresponding to all food materials in the first image, and stores the article identifier information contained in the first image in the control unit of the intelligent refrigerator. After the control unit stores the item identification information, the first image is displayed on a real-time food material shooting display area of the refrigerator, wherein in a specific implementation process, the item identification can refer to the name of the food material or the number of the food material and other information which uniquely represents the food material, for example, if an area in the first image shows an apple, the item identification corresponding to the area is the apple.

The process of identifying each food material in the first image belongs to the prior art, and in the embodiment of the present invention, the process is not described in detail.

Since the food material icon corresponding to the food material of each item identifier is also pre-stored in the food material database display area, when the control unit determines the item identifier corresponding to the food material added this time, the food material icon corresponding to the item identifier can be searched in the food material database display area, and the food material icon is displayed in the corresponding position of the virtual storage display area of the refrigerator. And when the control unit determines the food material taken out at this time, deleting the food material icon corresponding to the food material in the virtual storage display area.

Specifically, the control unit adds the food material icon to a virtual storage grid display interface of the refrigerator for display, may add the food material icon to an empty display position in a virtual storage display area of the food material management interface, where the empty display position may be a display position after the last displayed food material icon, and displays the food material icon on the display position of the virtual storage display area.

In the embodiment of the invention, the food materials added or taken out when the refrigerator door body is closed at the present time are determined based on the identification and comparison of the first image acquired at the present time and the second image acquired at the last time; and the food material icons corresponding to the food materials are added or deleted in the virtual storage display area, so that the management of the food materials is completed, and the efficiency of food material management is improved.

Example 3:

in order to better enhance the experience effect of the user, on the basis of the foregoing embodiments, in an embodiment of the present invention, the method further includes:

After the food material real-time shooting display area of the first image acquired by the image acquisition device 3 in the food material management interface is displayed, when a user needs to check more abundant food material information of a certain food material, for example, when the user wants to acquire and check menu information of a certain food material, the user can perform selection operation by shooting the display area in real time through the food material to acquire the food material information of the food material corresponding to the selection operation, so that the user experience is improved.

Specifically, because the control unit has previously stored the item identifier of the food material and the information of the area in the first image corresponding to the item identifier, when the control unit receives a selection operation of the user on the first image displayed on the real-time food material shooting display area, the selection area corresponding to the selection operation is determined, and according to the selection area, the item identifier corresponding to the selection area is searched from the item identifier of the food material and the information of the area corresponding to the item identifier, which are previously stored by the control unit.

The control unit can obtain the food material information of the food material from a local or server according to the article identification, and display the food material information after obtaining the food material information of the food material. Specifically, when the food material information of the food material is displayed, in a possible implementation manner, the food material management interface may further include a food material information display area, and the food material information of the food material is displayed in the food material information display area; in another possible implementation manner, the food material information of the food material is displayed in the virtual storage display area.

When the food material information of the food material selected by the user is acquired, in a possible implementation manner, the control unit locally can pre-store the food material information corresponding to each food material, after the user selects the food material in the first image on the real-time food material shooting display area, the control unit receives the selection operation of the user on the real-time food material shooting display area, determines the food material selected by the user, and searches the food material information of the food material selected by the user from the food material information corresponding to each food material pre-stored by the control unit.

In order to enable the control unit to more accurately determine the food material selected by the user, specifically, the selection operation may be a manual click operation, a press operation, or the like. When the selection operation performed by the user on the food real-time shooting display area is manual click operation, the area of the area selected by the click operation is small, the situation that a plurality of food materials are selected by one manual click operation cannot occur, and only one food material is selected in the area which is clicked. Therefore, according to the selection area manually clicked by the user, the control unit searches the item identifier of the food material corresponding to the selection area from the item identifier of the food material pre-stored by the control unit and the information of the area in the first image corresponding to the item identifier, and further can determine the food material selected by the user according to the item identifier.

When the selection operation of the user on the real-time food shooting display area is a pressing operation, the area selected by the pressing operation is usually relatively large, and a situation that a plurality of food materials in the first image are selected by one pressing operation may occur. When the selection operation is performed, the control unit may determine a selection area corresponding to the selection operation according to the selection operation, if the selection area includes the item identifiers of at least two food materials, according to an area corresponding to each item identifier in the first image stored by the control unit, the control unit identifies each area located in the selection area corresponding to the selection operation, identifies an area of each area located in the selection area corresponding to the selection operation, and determines an item identifier corresponding to an area with the largest area as the item identifier of the food material selected by the user.

For example, as shown in fig. 4, a user clicks a food material to shoot the food material on a display area in real time, and a control unit obtains a schematic diagram of food material information of the food material, where an upper portion of a display screen outside a refrigerator is the food material real-time shooting display area, and the food material real-time shooting display area specifically shows a first image of a three-layer shelf inside the refrigerator; the left side of the lower half part of the display screen is a virtual storage display area, and specifically, the virtual storage display area displays food material icons of food materials in the refrigerator; the right side of the lower half part of the display screen is a food material database display area, and the food material database display area displays a food material database.

Specifically, when a user performs a selection operation in a first image of the food material real-time shooting display area, the control unit determines a selection area selected by the selection operation, searches the item identifier of the food material corresponding to the selection area according to the pre-stored item identifier of the food material and the information of the area in the first image corresponding to the item identifier, and further determines the item identifier of the food material selected by the user according to the item identifier. According to the item identifier, the food material information of the food material can be acquired and displayed. Specifically, a possible implementation manner is that the control unit searches food material information of the item identifier from a server or from a local location according to the item identifier of the food material, where a food material information table of each item identifier is pre-stored in the server or the control unit, where the food material information table may include nutritional information of various food materials, and the like.

After the control unit determines the food material selected by the user, in order to provide better intelligent experience for the user, in the implementation of the present invention, after the food material information of the food material is acquired and displayed according to the item identifier, the method further includes:

Specifically, in order to show detailed information of the food material selected by the user to the user, for example, show the number of days that the food material has been put into the refrigerator to the user, when the user selects the food material on the first image in the real-time food material shooting display area, the control unit may highlight the food material icon of the food material in the virtual storage display area, so that the experience effect of the user is improved.

In an embodiment of the present invention, in order to better implement management of food materials, the food material information includes at least one of the following:

nutrition information, shelf life information and recipe information of the food materials.

The nutritional information of the food material refers to the efficacy of the food material, for example, the nutritional information of the yam can invigorate spleen, nourish stomach, help digestion, tonify lung, relieve cough and the like. The shelf life information refers to the optimal edible time of the food material. However, the shelf life information of the opened food will be shortened, so that the user can modify the shelf life information of the food material in the specific implementation process. For example, the shelf life information of the unopened yogurt is 21 days, but the shelf life information of the opened yogurt can be modified according to the selection of the user, for example, the shelf life information can be modified to 15 days. The recipe information refers to dishes made by the food material, for example, the recipe information of yam may include blueberry yam, purple potato yam, and the like.

The problem that a user does not know how to make dishes when having food materials or is greasy in a common way is solved. When a user carries out selection operation on a first image on a real-time food material shooting display area of the refrigerator, a selection area selected by the user is determined, related menu information of the food material is obtained according to an article identifier corresponding to the food material in the selection area, and the menu information is displayed in the food material information display area. In order to solve the problem that a user knows the nutritional information of the food material, so that the user can supplement related nutrition according to the nutritional information of the food material, the related nutritional information of the food material can be obtained, the shelf life information of the food material can be provided, and better food material management and intelligent experience are provided for the user.

Specifically, in a possible implementation manner, the control unit locally pre-stores recipes, a nutrition information table and a shelf life information table of various food materials, wherein the recipes include recipe information that certain food materials can be made, for example, the specific recipe information of the yam is the yam fried with sliced meat, the yam fried with clean fried broccoli and the like; the nutrition information table contains the nutritional effects of a certain food material, for example, the specific nutritional information of the yam is spleen strengthening, stomach tonifying, digestion aiding, lung benefiting, cough relieving and the like; the shelf life information table contains shelf life information of various food materials, for example, the shelf life information of yoghourt is 21 days, the shelf life information of pure milk is 180 days, and the like. When a user carries out selection operation on a first image displayed in the food material real-time shooting display area, according to an article identifier corresponding to a selection area selected by the user, related menu information, nutrition information and quality guarantee period information of the food material are obtained from a locally stored menu, a nutrition information table and a quality guarantee period information table, and the menu information, the nutrition information and the quality guarantee period information of the food material are displayed in the food material information display area.

In another possible implementation manner, the recipe, the nutritional information table and the shelf life information table are stored in a server, a control unit in the intelligent refrigerator sends item identification information corresponding to the food material to the server through a communication module in the control unit, and after the server receives the item identification, the server inquires related recipe information, nutritional information and shelf life information corresponding to the item identification. And the server sends the acquired related menu information, the acquired nutrition information and the acquired shelf life information to the control unit. And after the control unit receives the menu information, the nutrition information and the shelf life information corresponding to the article identification, displaying the related menu, the nutrition information and the shelf life information of the food material in a food material information display area.

Specifically, a first image to be displayed, namely, a food real-time shooting display area, a virtual storage display area, a food database display area, nutritional information, shelf life information and a recipe, can be simultaneously displayed on the display screen, so that the display screen can be divided into six areas according to the information to be displayed, for example, the six areas respectively display the food real-time shooting display area, the virtual storage display area, the food database display area and the food information display area. When a user performs selection operation on the real-time food material shooting display area, the recipe information, the nutrition information and the shelf life information of the food material are acquired from the control unit or the server according to the article identification of the food material selected by the user, wherein the recipe information of the food material is displayed in a recipe area in the food material information display area, the nutrition information of the food material is displayed in a nutrition information area in the food material information display area, and the shelf life information of the food material can be displayed in a shelf life area in the food material information display area.

Generally, related menu information of one food material is more, and in order to better display the menu information, preferably, a current interface of a display screen can display a food material real-time shooting display area, a virtual storage display area, a food material database display area and a food material information display area, wherein the food material information display area displays nutrition information and shelf life information of the food material. When a user wants to view menu information of a certain food material, the current food material management interface of the display screen can be switched to another interface, specifically, the current food material management interface can be switched to another interface through sliding operation of the user, and the interface is a menu interface. The recipe interface may display the first image and the recipe information. When the control unit receives a first image of a user on the food real-time shooting display area for selection operation, related menu information of the food is obtained from the control unit or the server according to the food selected by the user and displayed on a menu interface.

In order to simplify the content displayed on the current interface of the display screen, it is preferable that the nutritional information and the shelf life information of the food material can be displayed on the menu interface, so that the menu interface of the display screen can be divided into three regions according to the information to be displayed, for example, the menu interface of the display screen is divided into three regions, and the three regions respectively display the nutritional information, the shelf life information and the menu information. When the user performs selection operation on the first image, the recipe information, the nutrition information and the shelf life information of the food material are acquired from the control unit or the server according to the food material selected by the user, wherein the recipe information of the food material is displayed in the recipe area, the nutrition information of the food material is displayed in the nutrition information area, and the shelf life information of the food material can be displayed in the shelf life area.

For example, as shown in a menu interface of a display screen of a refrigerator in fig. 5, a display screen outside the refrigerator sequentially displays shelf life information, nutritional information and menu information of food materials from top to bottom, specifically, when a user clicks a yam in a first image on a real-time food material shooting display area, according to a food material corresponding to a selection area clicked by the user, related menu information, nutritional information and shelf life information of the food material are obtained from a control unit or a server, and related menu information, nutritional information and shelf life information of the yam are displayed on the menu interface, for example, the menu information of the yam includes stir-fried yam, blueberry yam, yam and radish soup; the quality guarantee period information of the Chinese yam is 3 days, and the quality guarantee period information of the Chinese yam and other food materials can be modified according to the actual requirements of users; the nutritional information of the Chinese yam is to strengthen the spleen and benefit the stomach and help digestion; benefiting lung and relieving cough, etc.

Example 4:

in order to enable the control unit to more accurately acquire food materials added or taken out by the user, on the basis of the above embodiments, in an embodiment of the present invention, the identifying and comparing the first image acquired this time with the second image acquired last time, and acquiring the food materials added or taken out by the user when the refrigerator door 2 is opened includes:

identifying each food material image in the first image and the second image, forming a plurality of first food material image identification frames in the first image, forming a plurality of second food material image identification frames in the second image, comparing any one first food material image identification frame with any one second food material image identification frame, and if the similarity between the first food material image identification frame and any one second food material image identification frame is smaller than a set threshold value, determining that the food material in the first food material image identification frame is the food material put in the door body 2 when the door body 2 is opened; comparing any one of the second food material image recognition frames with any one of the first food material image recognition frames, and if the similarity between the second food material image recognition frame and any one of the first food material image recognition frames is smaller than a set threshold value, the food material in the second food material image recognition frame is the food material taken out when the door body 2 is opened.

If the food material exists in the first image but cannot be found in the second image, determining that the food material is the food material added by the user; if the food material exists in the second image but cannot be found in the first image, it can be determined that the food material is the food material taken out by the user. Therefore, the food materials added or taken out by the user can be determined according to the food materials in the first image and the second image.

Specifically, the control unit prestores results of the second images after the image recognition operation, specifically, a plurality of second food material image recognition frames are obtained through the image recognition operation, after the first images are obtained, each food material image in the first images is recognized through the image recognition operation, and specifically, the first images obtain a plurality of first food material image recognition frames through the image recognition operation. For any first food material image recognition frame, similarity is respectively calculated with all second food material image recognition frames in the second images, and when all the obtained similarities are smaller than a set threshold value, the food material in the first food material image recognition frame is determined to be the food material put into by the user after the door body 2 is opened at this time; and for any second food material image recognition frame, respectively calculating the similarity with all the first food material image recognition frames in the first images, and when all the obtained similarities are smaller than a set threshold value, determining the food materials in the second food material image recognition frame as the food materials taken out by the user when the door body 2 is opened at this time by the control unit.

The process of calculating the similarity between the first food material image identification frame and the second food material image identification frame is the prior art, and in the embodiment of the invention, the process is not described in detail.

Example 5:

in order to achieve better management of target food materials put into a user and improve the experience effect of the user, on the basis of the above embodiments, in an embodiment of the present invention, the forming of the plurality of food material image recognition frames in the image includes:

Generally, when a user places food materials in a refrigerator, a plurality of food materials of the same type may be placed, for example, a plurality of yoghurts, apples, tomatoes, and the like may be placed at a time. For example, a user puts a plurality of apples at a time, and if each apple needs to be managed, the area corresponding to each apple can be selected respectively, so that the management of the plurality of apples is realized.

In order to facilitate the management of the food materials in the refrigerator by the user and improve the experience effect of the user, the same food materials which are generally put together for management may be put together. Therefore, the control unit in the smart refrigerator may merge regions in the image, and merge regions corresponding to the unmanaged food materials.

In order to better improve the experience effect of the user, after the control unit obtains the item identifier corresponding to each food material contained in the image, the areas corresponding to the food materials which are not managed in the image can be merged. Specifically, for each unmanaged area, when the item identifier of the food material corresponding to the area is different from the item identifier of the food material corresponding to the candidate area adjacent to the area, the candidate area may not be merged into the area; when the article identifier corresponding to the area is the same as the article identifier corresponding to the candidate area, whether the distance between the area and the candidate area is smaller than a preset distance threshold value or not needs to be judged, and if not, the candidate area does not need to be merged into the area; if so, merging the candidate region into the region, and the merged region is the food material image recognition frame corresponding to the food material identified by the item. That is, the food materials in the candidate area and the food materials in the area are considered to be the same food materials which are put in simultaneously, so that the food materials in the candidate area and the food materials in the area can be managed together. The distance between the two regions is judged, the minimum distance between the regions can be identified, the minimum distance between the regions is determined as the distance between the two regions, for example, the two regions are respectively a left region and a right region, the distance between the rightmost pixel point of the left region and the leftmost pixel point of the right region can be identified, and the distance between the two regions is determined according to the distance.

Preferably, the control unit takes the areas corresponding to the unmanaged food materials on the leftmost side (left and right sides in the image) as the starting points, sequentially moves to the right, calculates the distance between the area and the candidate area when recognizing that the item identifier corresponding to the food material of the candidate area adjacent to the area on the right is still the same as the item identifier corresponding to the food material of the area, and judges that the food material of the area and the food material of the candidate area are the same food material put in at the same time when the distance between the area and the candidate area is smaller than a preset distance threshold value, and merges the candidate areas into the area. And then sequentially rightwards, when different food materials are identified, the positions of the identified food materials of the same kind are divided into an area, and the combined area is used as a food material image identification frame corresponding to the food material of the article identifier.

For example, after 4 apples are placed in a refrigerator, the article identifier of the food material in the image of each apple and the information of the area in the image corresponding to the article identifier are obtained according to the image recognition technology, the control unit extends rightward (leftward and rightward in the image) with the recognized leftmost area (leftward and rightward in the image) as a starting point, calculates the distance between the area and the candidate area if the article identifier of the candidate area adjacent to the area is the same as the article identifier of the area, and merges the candidate area into the area if the distance is smaller than a preset distance threshold. Similarly, continuing to judge to the right, if the article identifier of the candidate area adjacent to the area is identified to be still the same as the article identifier of the area, calculating the distance between the area and the candidate area, if the distance is smaller than a preset distance threshold, merging the candidate area into the area, and if the article identifier of the adjacent area is different from the article identifier of the area, taking the candidate area as a new starting point, and performing the above operation. If the recognized distance between the same food materials is larger than a preset distance threshold value, the food materials are considered to be the same food materials put in at different time, the food materials are divided into two areas respectively, and two icons of the food materials are added for distinguishing and managing. And performing distance calculation and area division on other food materials in the same way, and taking the combined area as a food material image recognition frame corresponding to the food material of the item identifier.

Example 6:

in order to better manage target food materials put in by a user and distinguish the same food materials put in different shelves or the same shelf at different times so as to improve the user experience, on the basis of the above embodiments, in an embodiment of the present invention, displaying food material icons of the food materials added this time in corresponding positions of a virtual storage display area of the refrigerator includes:

for the food materials added when the door body 2 is opened at this time, determining the information of the target shelf of the food materials in the refrigerator according to the position information of the food materials in the first image; and displaying the food material icon added this time in a display area of the virtual storage display area corresponding to the target shelf.

Because the position of the camera arranged in the image acquisition device 3 in the refrigerator is fixed, each pixel point in the image acquired by the camera corresponds to the actual physical position in the refrigerator, and for the food materials added by the user when the refrigerator door body 2 is opened, the information of the target shelf of the food materials in the refrigerator can be determined according to the position information of the food materials in the first image. And adding a food material icon corresponding to the food material to a display area of the target shelf corresponding to a virtual storage display area of the refrigerator to display the food material icon according to the information of the target shelf of the food material.

Each shelf of a refrigerating chamber of the intelligent refrigerator is provided with a corresponding display area in the virtual storage display area, and after the target shelf information where the food material added at this time is located is identified, the food material icon corresponding to the food material is added to the display area of the virtual storage display area corresponding to the target shelf information. Specifically, when the food material icons are added to the display area corresponding to the shelf, the food material icons may be sequentially added according to the order in which the food materials are added to the shelf.

Fig. 4 is a schematic diagram of a display screen outside a refrigerator, in which the upper half portion (upper and lower portions of the image) of the display screen shows a real-time food shooting display area, and the real-time food shooting display area displays a first image of three shelves inside the refrigerator shot by a camera; the left side of the lower half part of the intelligent refrigerator is displayed with a virtual storage display area of the refrigerator, wherein each shelf of the intelligent refrigerator is provided with a corresponding display area in the virtual storage display area, food material icons of food materials stored in a refrigerating chamber of the refrigerator are displayed, and related information of the food materials, such as storage dates of the food materials in the refrigerator, can be added to the food material icons; the right side of the lower half part is shown as a food material database display area, and a food material database is shown in the food material database display area. The first image displayed on the upper half part of the display screen is formed by splicing images collected by a plurality of cameras, and each pixel point in the first image corresponds to the actual physical position in the refrigerator, so that the information of a target shelf of food materials in the refrigerator can be determined according to the position information of the food materials added at this time in the first image. Therefore, the food material icon of the food material can be displayed in the display area of the virtual storage display area of the refrigerator corresponding to the target shelf.

In order to prevent the food materials from being overdue due to the fact that the user puts the food materials into the refrigerator for too long time, in the embodiment of the present invention, the method further includes:

The food in the refrigerator is better managed, and the situation that the food is deteriorated due to the fact that a user forgets the days when the food is put into the refrigerator after the user puts the food into the refrigerator is prevented. Therefore, after the food material is put into the refrigerator, the time length of the food material put into the refrigerator needs to be displayed in the virtual storage display area of the refrigerator.

Specifically, after food material icons of food materials added when the door body 2 is opened are added to a virtual storage display area of the refrigerator, first initial time when the food material icons are added needs to be recorded, and time length of the food materials corresponding to each food material icon being put in is determined according to a difference value between the current time and the first initial time, wherein the time length is generally recorded in days. And after the duration is obtained, displaying the duration at the position corresponding to the food material icon in the virtual storage display area of the refrigerator. In a specific implementation, information of a target shelf of the food material icon can be displayed.

Specifically, the duration and the information of the target shelf of the food material icon are displayed at the position corresponding to the food material icon in the virtual storage display area of the refrigerator, and may be displayed at a vacant position adjacent to the food material icon or displayed on the food material icon, preferably, in order to make the display of the virtual storage display area of the refrigerator more beautiful, the duration is displayed at the upper right corner of the food material icon, and the information of the target shelf of the food material icon is displayed at the lower right corner of the food material icon; or the duration may be displayed at the lower right corner of the food material icon, and the information of the target shelf of the food material icon is displayed at the upper right corner of the food material icon.

Specifically, as shown in fig. 6, the upper portion of the display screen outside the refrigerator shows a real-time food shooting display area; the left side of the lower half part of the display screen displays a virtual storage display area of the refrigerator; and the right side of the lower half part of the display screen is displayed as a food material database display area. The duration corresponding to the food material icon can be displayed at the upper right corner of the food material icon, and the information of the target shelf corresponding to the food material icon can be displayed at the lower right corner of the food material icon.

In order to better implement management of food materials stored in a refrigerator, in an embodiment of the present invention, the method further includes:

In order to better improve the experience effect of the user, the control unit may preset a good duration management range. Specifically, the set time length management ranges are not overlapped, and different font colors are corresponding to different time length management ranges. Specifically, for the duration corresponding to each food material image, the duration management range corresponding to the duration is determined according to each preset duration management range, and the duration is displayed at the position corresponding to the food material icon by adopting the font color corresponding to the duration management range. For example, the time length management range is divided into three ranges, namely a first time length management range, a second time length management range and a third time length management range. The first time-long physical range can be 0-3 days, and the font color corresponding to the first time-long physical range is green; the second time length management range can be 4-6 days, and the font color corresponding to the second time length management range is yellow; the third duration management range may be 7 days or more, and a font color corresponding to the third duration management range is red.

For example, fig. 7 is a schematic diagram of an external display screen of a refrigerator, in which the upper half (upper and lower sides of the image) of the display screen displays a first image of three shelves inside the refrigerator, the first image being captured by a camera; the left side of the lower half part of the intelligent refrigerator is displayed with a virtual storage display area of the refrigerator, wherein each shelf of the intelligent refrigerator is provided with a corresponding display area in the virtual storage display area, and food material icons of food materials stored in a refrigerating chamber of the refrigerator are displayed; the right side of the lower half portion shows a food material database display area. For example, if the last food material broccoli in the second row in the first image displayed in the upper half portion of fig. 5 is stored for 4 days, that is, the time period for which the food material is managed is 4 days, the time period displayed at the position corresponding to the food material icon in the virtual storage display area of the refrigerator is yellow; if the second food material banana in the first row in the first image displayed in the upper half portion of fig. 5 is stored for 3 days, that is, the time duration for which the food material is managed is 3 days, the time duration displayed at the position corresponding to the food material icon in the virtual storage display area of the refrigerator is green; if the fourth food material ham in the first row in the first image shown in the upper half of fig. 3 has been stored for 7 days, i.e., the time period during which the food material is managed is 7 days, the time period displayed at the position corresponding to the food material icon in the virtual storage display area of the refrigerator is red.

For example, the flow diagram of the food material management method of the intelligent refrigerator shown in fig. 8 mainly includes the following steps:

s801: after the user puts the food material into the refrigerator, the refrigerator door is closed.

S802: when the control unit identifies that the refrigerator door is closed, a first image of food materials in the refrigerator is acquired through the image acquisition device 3, image identification operation is carried out on the first image, an article identifier corresponding to each food material contained in the image is obtained, and the first image is displayed on a food material real-time shooting display area of the refrigerator.

S803: the control unit acquires the article identifier of the food material added or taken out by the user when the refrigerator door body 2 is closed according to the article identifier corresponding to each food material and the article identifier corresponding to each food material in the second image, displays the food material icon of the food material added at this time at the corresponding position of the virtual storage display area of the refrigerator, and deletes the food material icon of the food material taken out at this time in the virtual storage display area.

S804: adding the food material icon corresponding to the food material into a virtual storage display area, calculating the managed time length of the food material, determining the information of a target shelf where the food material is located according to the position information of the food material, and displaying the food material icon, the time length corresponding to the food material icon and the information of the target shelf in the virtual storage display area.

S805: and the user performs selection operation on the first image to determine the food material selected by the user.

S806: when the control unit receives a selection operation of a user on the first image, the food material icon of the food material selected by the user is highlighted in the virtual storage display area of the refrigerator.

S807: when the control unit receives a selection operation of a user on the first image, related menu information, nutrition information and shelf life information of the food material are displayed in a food material information display area of the refrigerator.

In conclusion, the live food materials in the refrigerator are displayed on the external display screen of the refrigerator, services such as expiration reminding of each food material are provided for a user, and management of the food materials in the refrigerator is achieved.

Example 7:

fig. 9 is a food material management device of an intelligent refrigerator according to an embodiment of the present invention, where the device includes:

the control module 901 is configured to control an image acquisition device to acquire a first image containing food materials in the refrigerator after the detected door of the refrigerator is closed, wherein the image acquisition device is configured to capture and acquire the image containing the food materials in the refrigerator;

the management module 902 is configured to identify and compare the first image obtained this time with the second image obtained last time, obtain the food material added or taken out when the door body is opened this time, display a food material icon of the food material added this time at a corresponding position of the virtual storage display area of the refrigerator, and delete the food material icon of the food material taken out this time in the virtual storage display area.

The management module 902 is further configured to determine the food material selected by the user, acquire and display food material information of the food material when receiving a selection operation of the user on the real-time food material shooting display area.

Further, the management module is further configured to highlight the food material icon corresponding to the food material selected by the user in the virtual storage display area.

The management module 902 is specifically configured to identify each food material image in the first image and the second image, form a plurality of first food material image identification frames in the first image, form a plurality of second food material image identification frames in the second image, compare any one of the first food material image identification frames with any one of the second food material image identification frames, and if the similarity between the first food material image identification frame and any one of the second food material image identification frames is smaller than a set threshold, determine that the food material in the first food material image identification frame is the food material put in when the door body is opened this time; comparing any one of the second food material image recognition frames with the first food material image recognition frame, and if the similarity between the second food material image recognition frame and any one of the first food material image recognition frames is smaller than a set threshold value, the food material in the second food material image recognition frame is the food material taken out by opening the door body.

The management module 902 is specifically configured to, for each region in the image, determine whether the article identifiers of the region and the food materials corresponding to candidate regions adjacent to the region are the same according to the article identifier corresponding to each region; if the distance between the area and the candidate area is the same as the preset distance threshold, judging whether the distance between the area and the candidate area is smaller than the preset distance threshold; if so, merging the candidate region into the region, and taking the merged region as a food material image recognition frame corresponding to the food material of the item identifier.

The management module 902 is specifically configured to determine information of a target shelf of food materials in the refrigerator according to position information of the food materials in the first image for the food materials added when the door body is opened at this time; and displaying the food material icon added this time in a display area of the virtual storage display area corresponding to the target shelf.

The management module 902 is further configured to determine a duration for which the food material corresponding to each food material icon is managed according to the stored first initial time and current time for which each managed food material is managed; and displaying corresponding duration at the corresponding position of each food material icon in the virtual storage display area.

The management module 902 is further configured to determine, for the determined duration corresponding to each food material icon, a duration management range matched with the duration according to each preset duration management range, and display the duration at a position corresponding to the food material icon in the virtual storage display area by using a font color corresponding to the duration management range.

Example 8:

on the basis of the foregoing embodiments, an embodiment of the present invention further provides an intelligent refrigerator 1000, as shown in fig. 10, including: the system comprises a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002 and the memory 1003 are communicated with each other through the communication bus 1004;

the memory 1003 has stored therein a computer program which, when executed by the processor 1001, causes the processor 1001 to perform the steps of:

The communication bus mentioned in the above-mentioned smart refrigerator may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.

The communication interface 1002 is used for communication between the above-described intelligent refrigerator and other devices.

The processor 1001 further performs the following steps:

The processor 1001 is further configured to highlight the food material icon corresponding to the food material selected by the user in the virtual storage display area.

The processor 1001 is further configured to identify each food material image in the first image and the second image, form a plurality of first food material image identification frames in the first image, form a plurality of second food material image identification frames in the second image, compare any one of the first food material image identification frames with any one of the second food material image identification frames, and if the similarity between the first food material image identification frame and any one of the second food material image identification frames is smaller than a set threshold, determine that the food material in the first food material image identification frame is the food material put in when the door body is opened this time; comparing any one of the second food material image recognition frames with the first food material image recognition frame, and if the similarity between the second food material image recognition frame and any one of the first food material image recognition frames is smaller than a set threshold value, the food material in the second food material image recognition frame is the food material taken out by opening the door body.

The processor 1001 is further configured to, for each region in the image, determine whether the article identifiers of the region and the food material corresponding to the candidate region adjacent to the region are the same according to the article identifier corresponding to each region; if the distance between the area and the candidate area is the same as the preset distance threshold, judging whether the distance between the area and the candidate area is smaller than the preset distance threshold; if so, merging the candidate region into the region, and taking the merged region as a food material image recognition frame corresponding to the food material of the item identifier.

The processor 1001 is further configured to determine information of a target shelf of the food material in the refrigerator according to the position information of the food material in the first image for the food material added by opening the door body this time; and displaying the food material icon added this time in a display area of the virtual storage display area corresponding to the target shelf.

The processor 1001 is further configured to determine a duration for which the food material corresponding to each food material icon is managed according to the stored first initial time and current time for which each managed food material is managed; and displaying corresponding duration at the corresponding position of each food material icon in the virtual storage display area.

The processor 1001 is further configured to determine, for the determined duration corresponding to each food material icon, a duration management range matched with the duration according to each preset duration management range, and display the duration at a position in the virtual storage display area corresponding to the food material icon by using a font color corresponding to the duration management range.

The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Alternatively, the memory may be at least one memory device located remotely from the processor.

The processor may be a general-purpose processor, including a central processing unit, a Network Processor (NP), and the like; but may also be a Digital instruction processor (DSP), an application specific integrated circuit, a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like.

In the embodiment of the invention, when the processor executes the program stored in the memory, the identification and comparison based on the first image acquired this time and the second image acquired last time are realized, and the food material added or taken out when the door is closed this time is determined; and the food material icons corresponding to the food materials are added or deleted in the virtual storage display area, so that the management of the food materials is completed, and the efficiency of food material management is improved.

Example 9:

on the basis of the foregoing embodiments, an embodiment of the present invention further provides a computer storage readable storage medium, in which a computer program executable by an intelligent refrigerator is stored, and when the program runs on the intelligent refrigerator, the intelligent refrigerator implements the following steps when executed:

The method further comprises the following steps:

The food material information comprises at least one of the following:

nutrition information, shelf life information, recipe information.

The identifying and comparing the first image obtained this time with the second image obtained last time to obtain the food material added or taken out when the door body is opened comprises the following steps:

The forming of the plurality of food material image recognition frames in the image comprises:

The step of displaying the food material icons of the food materials added at this time in the corresponding positions of the virtual storage display area of the refrigerator comprises the following steps:

The method further comprises the following steps:

The computer readable storage medium may be any available medium or data storage device that can be accessed by a processor in a smart refrigerator, including but not limited to magnetic memory such as a flexible disk, a hard disk, magnetic tape, a magneto-optical disk (MO), etc., optical memory such as a CD, a DVD, a BD, an HVD, etc., and semiconductor memory such as a ROM, an EPROM, an EEPROM, a non-volatile memory (NANDFLASH), a Solid State Disk (SSD), etc.

In the computer-readable storage medium provided in the embodiment of the present invention, a computer program is stored, and when the computer program is executed by a processor, the identification and comparison between a first image obtained this time and a second image obtained last time are realized, and a food material added or taken out this time is determined; and the food material icons corresponding to the food materials are added or deleted in the virtual storage display area, so that the management of the food materials is completed, and the efficiency of food material management is improved.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.

Claims

1. An intelligent refrigerator comprising: the storage box is arranged on the door body; it is characterized by also comprising an image acquisition device, a display device and a control unit,

2. The intelligent refrigerator according to claim 1, wherein the control unit is further configured to determine the food material selected by the user and obtain food material information of the food material when receiving a selection operation of the user on the real-time food material shooting display area;

the display device is also used for displaying the food material information.

3. The intelligent refrigerator according to claim 2, wherein the control unit is further configured to highlight a food material icon corresponding to the food material selected by the user in the virtual storage display area when receiving a selection operation of the user on the real-time food material shooting display area.

4. The intelligent refrigerator of claim 2, wherein the food material information comprises at least one of:

nutrition information, shelf life information, recipe information.

5. The intelligent refrigerator as claimed in claim 2, wherein the display device further comprises a food material database display area for displaying icons of various food materials.

6. The intelligent refrigerator according to claim 1, wherein the control unit is specifically configured to identify each food material image in the first image and the second image, form a plurality of first food material image identification frames in the first image, and form a plurality of second food material image identification frames in the second image; comparing any one first food material image recognition frame with any one second food material image recognition frame, and if the similarity between the first food material image recognition frame and any one second food material image recognition frame is smaller than a set threshold value, determining that the food materials in the first food material image recognition frame are the food materials put in the door body when the door body is opened; comparing any one of the second food material image recognition frames with the first food material image recognition frame, and if the similarity between the second food material image recognition frame and any one of the first food material image recognition frames is smaller than a set threshold value, the food material in the second food material image recognition frame is the food material taken out by opening the door body.

7. The intelligent refrigerator according to claim 6, wherein the control unit is further configured to determine, for each area in the first image, whether the item identifiers of the area and the food material corresponding to the candidate area adjacent to the area are the same according to the item identifier corresponding to each area; if the distance between the area and the candidate area is the same as the preset distance threshold, judging whether the distance between the area and the candidate area is smaller than the preset distance threshold; if so, merging the candidate region into the region, and taking the merged region as a food material image recognition frame corresponding to the food material of the item identifier.

8. The intelligent refrigerator according to claim 1, wherein the control unit is specifically configured to determine information of a target shelf of the food material in the refrigerator according to the position information of the food material in the first image for the food material added at the current door opening; and displaying the food material icon added this time in a display area of the virtual storage display area corresponding to the target shelf.

9. The intelligent refrigerator according to claim 1, wherein the control unit is further configured to determine a time duration for which the food material corresponding to each food material icon is managed according to the stored first initial time and current time for which each managed food material is managed; and displaying corresponding duration at the corresponding position of each food material icon in the virtual storage display area.

10. The intelligent refrigerator according to claim 9, wherein the control unit is further configured to determine, for each determined duration corresponding to each food material icon, a duration management range matching the duration according to each preset duration management range, and display the duration at a position corresponding to the food material icon in the virtual storage display area by using a font color corresponding to the duration management range.

11. A food material management method of an intelligent refrigerator is characterized by comprising the following steps:

12. The method of claim 11, wherein the method further comprises:

13. The method of claim 12, wherein the method further comprises:

14. The method of claim 12, wherein the food material information comprises at least one of:

nutrition information, shelf life information, recipe information.

15. The method of claim 11, wherein the identifying and comparing the first image obtained this time with the second image obtained last time to obtain the food material added or taken out when the door body is opened this time comprises:

16. The method of claim 15, wherein forming a plurality of food material image recognition boxes within the image comprises:

17. The method of claim 11, wherein the displaying the food material icons of the currently added food materials at corresponding positions in the virtual storage display area of the refrigerator comprises:

18. The method of claim 11, wherein the method further comprises:

19. The method of claim 18, wherein the method further comprises:

20. An edible material management apparatus of an intelligent refrigerator, the apparatus comprising:

21. An intelligent refrigerator, comprising: the system comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

the memory has stored therein a computer program which, when executed by the processor, causes the processor to carry out the steps of the method of any of claims 11 to 19.

22. A computer-readable storage medium, characterized in that it stores a computer program executable by an intelligent refrigerator, which when run on the intelligent refrigerator causes the intelligent refrigerator to perform the steps of the method of any one of claims 11 to 19.