WO2023124088A1

WO2023124088A1 - Method for identifying information of articles in refrigerator, and refrigerator

Info

Publication number: WO2023124088A1
Application number: PCT/CN2022/110222
Authority: WO
Inventors: 高洪波; 孔令磊
Original assignee: 青岛海尔电冰箱有限公司; 海尔智家股份有限公司
Priority date: 2021-12-27
Filing date: 2022-08-04
Publication date: 2023-07-06
Also published as: CN114862867A

Abstract

Disclosed in the present invention are a method for identifying information of articles in a refrigerator, and the refrigerator. The method comprises: segmenting an obtained aerial view image into a plurality of detection areas, and identifying whether edge information of a preset shape exists in the detection areas or not; if so, calculating the proportion of pixel points of a preset color and pixel points of other colors in the detection areas; and if the proportion of the pixel points of the preset color and the pixel points of the other colors in the detection areas is within a set threshold range, positioning the pixel points of the preset color, and identifying text information according to the pixel points of the preset color. According to the present invention, the specific location of text on an article can be quickly positioned, the text information is identified, the calculation amount is small, and any text information to be identified is not missed.

Description

Method for identifying item information in refrigerator and refrigerator

This application claims the priority of a Chinese patent application with an application date of December 27, 2021, an application number of 202111614041.6, and an invention titled "method for identifying item information in a refrigerator, refrigerator and computer storage medium", the entire contents of which are incorporated by reference in this application.

technical field

The invention relates to the field of refrigeration devices, in particular to a method for identifying item information in a refrigerator and a refrigerator.

Background technique

With the development of intelligent electrical appliances, the identification of internal items has become a necessary function of smart refrigerators. Usually, in order to realize the identification of the items stored in the refrigerator, one or more cameras are installed inside the refrigerator to collect photos of the items in the refrigerator, and the collected photos are identified and processed and then sent to the user terminal for viewing by the user. Setting the camera above the items can avoid the problem of items occluding each other, and improve the accuracy of item information identification.

However, there are still many problems in the specific implementation of the smart refrigerator, for example, how to identify the text information of the items in the refrigerator bottle holder, including the item name, production date, shelf life and so on. The existing method is to substitute the image of the inner space of the bottle seat of the refrigerator into the algorithm, locate the specific position of the text information of the item on the image, and then identify the text information of the item. This method has a large amount of calculation, and it is easy to locate inaccurate or missing text information.

Contents of the invention

The object of the present invention is to provide a method for identifying item information in a refrigerator and a refrigerator.

In order to achieve one of the purposes of the above invention, an embodiment of the present invention provides a method for identifying information on items in a refrigerator, including steps:

Obtain a bird's-eye view image of the space inside the refrigerator bottle holder;

dividing the bird's-eye view image into a plurality of detection areas;

identifying whether there is edge information of a preset shape in the detection area;

If there is edge information of a preset shape, calculating the ratio of the preset color pixel and other color pixels in the detection area;

If the proportion of the pixel of the preset color and the pixel of other colors in the detection area is within the set threshold range, the pixel of the preset color is located, and text information is identified according to the pixel of the preset color.

As a further improvement of an embodiment of the present invention, "segmenting the bird's-eye view image into multiple detection areas" specifically includes:

The detection area is a square structure;

The detection area is at least larger than the area of the top cover of the item in the bottle holder of the refrigerator;

There is a partial overlap between every two adjacent detection areas.

As a further improvement of an embodiment of the present invention, "there is partial overlap between every two adjacent detection areas" is specifically:

There is a partial overlapping area between every two adjacent detection areas, and the overlapping area is half of the detection area.

As a further improvement of an embodiment of the present invention, "identifying whether there is edge information of a preset shape in the detection area" specifically includes:

Using a circle detection algorithm and/or a square detection algorithm to identify whether there is circular edge information and/or square edge information in the detection area.

As a further improvement of an embodiment of the present invention, "calculating the ratio of preset color pixels and other color pixels in the detection area" specifically includes:

Acquire all preset color pixels in the detection area;

The preset color is specifically black.

As a further improvement of an embodiment of the present invention, "if the proportion of the preset color pixel and other color pixels in the detection area is within the set threshold range, locate the preset color pixel, and The aforementioned preset color pixel point recognition text information" specifically includes:

The threshold range is specifically 30% to 40%;

An OCR algorithm is used to locate the specific position of the pixel of the preset color, and the text information is recognized according to the pixel of the preset color.

As a further improvement of an embodiment of the present invention, the method includes:

Identify whether there is edge information of a preset shape in each detection area one by one.

As a further improvement of an embodiment of the present invention, the method also includes:

If there is a preset shape in the detection area, calculate the proportion of the preset color pixel and other color pixels in the detection area;

If the proportion of the preset color pixel and other color pixels in the detection area is within the set threshold range, the preset color pixel is located, and the text information is identified according to the preset color pixel.

If there is no edge information of a preset shape in the detection area, continue to identify the next detection area;

If the ratio of preset color pixels and other color pixels in the detection area is not within the set threshold range, continue to identify the next detection area.

One embodiment of the present invention provides a refrigerator, including: a bottle holder, a camera, a memory, and a processor, wherein,

The camera is located at the middle position on the top of the bottle seat and is set vertically downward;

The bottom of the bottle seat is used for placing articles;

The memory stores a computer program that can run on the processor, and the processor implements the steps of the method for identifying item information in the refrigerator as described in any one of the above implementation manners when executing the program.

Compared with the prior art, the present invention divides the acquired bird's-eye view image into a plurality of detection areas, by identifying whether there is a preset shape in each detection area and calculating the preset color pixel points and other color pixel points in each detection area It can quickly locate the specific position of the text on the item and recognize the text information. The amount of calculation is small, and any text information that needs to be recognized will not be missed.

Description of drawings

Fig. 1 is a schematic flowchart of a method for identifying information on items in a refrigerator in an embodiment of the present invention.

Fig. 2 is a schematic diagram of a bird's-eye view captured in an embodiment of the present invention (adjacent detection areas do not overlap).

Fig. 3 is a schematic diagram of a bird's-eye view captured in an embodiment of the present invention (adjacent detection areas partially overlap).

Fig. 4 is a schematic flowchart of a method for identifying item information in a refrigerator in another embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the following will clearly and completely describe the technical solution of the present application in combination with specific implementation methods of the present application and corresponding drawings. Apparently, the described implementations are only some of the implementations of this application, not all of them. Based on the implementation manners in this application, all other implementation manners obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

For the convenience of description, terms representing relative positions in space are used herein for description, such as "upper", "lower", "rear", "front", etc., which are used to describe the relative position of one unit or feature shown in the drawings relative to another. A unit or feature relationship. Spatially relative terms may encompass different orientations of the device in use or operation other than the orientation shown in the figures. For example, if the device in the figures is turned over, elements described as "below" or "above" other elements or features would then be oriented "below" or "above" the other elements or features. Thus, the exemplary term "below" can encompass both a spatial orientation of below and above.

Since the location of the information such as the production date and shelf life of the ingredients in the refrigerator is scattered and not fixed, and some are even set at the bottom of the ingredients (for example, the production date of some yogurt is set at the bottom), it is inconvenient to identify and manage these ingredients in a unified manner, so When the ingredients are stored in the refrigerator, a label in a uniform format containing the basic information of the ingredients can be printed for it, and the label can be placed at a fixed position of the ingredients (for example, pasted on the top of the ingredients), so that the computer and the user can obtain information through the labels. Information such as the name and production date of the ingredients is convenient for computers and users to manage the ingredients.

As shown in Figure 1, an embodiment of the present invention provides a method for identifying item information in a refrigerator, including steps:

S1: Obtain a bird's-eye view image 1 of the space inside the refrigerator bottle holder.

In this embodiment, the camera is set on the top of the bottle seat, and the object is photographed from above to obtain a bird's-eye view image. In order to avoid mutual occlusion between the items in the bottle seat, which is not conducive to the detection and identification of item information, the present invention preferably uses the camera It is located in the middle of the top of the bottle holder.

Of course, in other embodiments of the present invention, two or more cameras can also be set in the bottle holder to obtain two or more bird's-eye images of the space in the refrigerator bottle holder taken from different angles, or The camera is arranged at the bottom of the bottle seat, and is set vertically downward, which is used to capture the information of the items in the bottle seat below it.

S2: Divide the bird's-eye view image into a plurality of detection areas 2 .

Exemplarily, as shown in FIG. 2 , which is a schematic diagram of a bird's-eye view image 1 captured in this embodiment, the bird's-eye view image 1 is divided into multiple detection areas 2 . In one embodiment of the present invention, the detection area 2 has a square structure, and the detection area 2 is at least larger than the area of the top cover of the item in the bottle holder of the refrigerator.

Specifically, in practical applications, the area of the top cover of the item placed in the bottle holder of the refrigerator generally does not exceed 400 pixels×400 pixels, so the size of the detection area in this embodiment is set to 400 pixels×400 pixels. Of course, in other embodiments of the present invention, the detection area 2 may also be in any other shape, as long as the bird's-eye view image 1 can be segmented as a whole for subsequent detection and recognition.

Further, there is a partial overlap between every two adjacent detection areas 2, and the size of the overlapping area is related to the detection accuracy set by the system. The larger the overlapping area of two adjacent detection areas 2, the greater the detection accuracy and the finer the recognition . In this embodiment, the overlapping area between every adjacent two detection areas 2 is set as half the size of the detection area 2, that is, the width of the overlapping area is 200 pixels, as shown in Figure 3, while improving the detection accuracy, it is also It will not increase the calculation amount of the system.

S3: Identify whether there is preset shape edge information in the detection area 2 .

Items that are usually placed in the bottle holder of the refrigerator can be roughly classified into bottled items, canned items, and boxed items. Among them, bottled items have bottle caps, which are items with obvious bottle caps, such as bottled beverages. , the outer contour of the top view of the bottle cap is circular; the outer contour of the top view of canned goods is circular, such as canned beverages, seasoning sauces and other items whose outer packaging is basically cylindrical; the top view shape of boxed goods is similar to Square structure, such as boxed milk products and other items whose outer packaging is basically cuboid or cube.

In order to facilitate the identification and unified management of the items and ingredients in the bottle holder of the refrigerator, the item type information or the item production date, shelf life and other information are generally printed on the top cover of the item. For the above-mentioned bottled items, canned items and boxed items The outer contour of the top view of the article can be divided into two categories: circular and square. In one embodiment of the present invention, the preset shape can be set as a circular and/or square structure, specifically identifying whether there is edge information of a circular and/or square structure in the detection area 2 .

Specifically, the circular detection algorithm and/or square detection algorithm can be used to identify whether there is circular edge information and/or square edge information in all the detection areas segmented in the bird's-eye view image. Here, the circular detection algorithm and the square detection algorithm All are prior art, and will not be repeated here.

S4: If there is a preset shape, calculate the proportion of the pixel of the preset color and the pixel of other colors in the detection area 2 .

Before calculating the ratio of the preset color pixel points and other color pixel points in the detection area 2, a step is further included: acquiring all preset color pixel points in the detection area 2 .

Since the text information such as item type information, production date, and shelf life printed on the item is usually in black font, so in one embodiment of the present invention, the preset color is specifically black, that is, the detection area 2 with a preset shape is calculated. The proportion of black pixels and other color pixels in the image.

S5: If the proportion of the preset color pixel point and other color pixel points in the detection area 2 is within the set threshold range, locate the preset color pixel point, and identify the preset color pixel point according to the preset color pixel point described text information.

Specifically, in the refrigerator bottle holder, it is impossible that only the text information on the upper surface of the item is a black pixel part, because in actual situations, there may be a large area of black area on the bottom of the bottle holder or on the wall of the bottle holder due to light blocking, and the item There are still some gaps between the characters on the top cover, so the proportion of black pixels and other color pixels in the detection area 2 must be within a threshold range. In an embodiment of the present invention, the threshold range is specifically Set it to 30% to 40% to eliminate the interference of the recognized black area except the text information on the top cover of the item, so as to accurately locate the area with text information.

Further, in this embodiment, the detection area where the edge information of the preset shape is identified and the proportion of black pixels and pixels of other colors is within 30% to 40% is obtained, and the black pixels are located using the OCR algorithm The specific position of the point, and recognize the specific text information according to the black pixel point located.

An embodiment of the present invention also provides another method for detecting and identifying item information in a refrigerator, which identifies whether there is edge information of a preset shape in each detection area 2 one by one. The specific detection and identification method is shown in Figure 4, and the specific steps include:

S2: Divide the bird's-eye view image 1 into multiple detection areas 2 .

Similarly, in one embodiment of the present invention, the detection area 2 has a square structure, which is at least larger than the top cover area of the items in the refrigerator bottle holder. Specifically, the detection area 2 has a range of 400 pixels×400 pixels. There is a partial overlap between every two adjacent detection areas, and the overlapping area is half of the detection area.

S3': Identify whether there is edge information of a preset shape in each detection area 2 one by one.

In this embodiment, the preset shape is specifically a circle and/or a square, and a circle detection algorithm and/or a square detection algorithm is used to identify whether there is circular edge information and/or a square edge in the detection area 2 information.

If edge information of a preset shape exists in the detection area 2, step S4' is performed; if edge information of a preset shape does not exist in the detection area 2, step S6 is performed.

S4': Calculate the ratio of preset color pixels and other color pixels in the detection area 2.

In this embodiment, the preset color is specifically black, and it is calculated whether the proportion of black pixels and pixels of other colors in the detection area 2 is within the set threshold range, specifically, the threshold range is specifically set as 30% to 40%, to eliminate the interference of the recognized black area except the text information on the top cover of the item, and accurately locate the area where text information exists.

If the proportion of black pixels and other color pixels in the detection area 2 is within the set threshold range, then execute S5'; if the proportion of black pixels and other color pixels in the detection area 2 is not If it is within the set threshold range, execute S6.

S5': Locate the preset color pixels, and identify text information according to the preset color pixels.

Specifically, in this embodiment, an OCR algorithm is used to locate the specific position of the black pixel, and specific text information is identified according to the located black pixel.

S6: continue to identify the next detection area 2 .

The present invention also provides a refrigerator, which includes a bottle holder, a camera, a memory and a processor.

Further, the bottle seat includes a top, a bottom, and a wall of the bottle seat, and the camera is set at the middle of the top of the bottle seat and is set vertically downward.

The bottom of the bottle seat is used to place items, and the memory stores a computer program that can run on the processor. When the processor executes the program, the method for identifying item information in the refrigerator in any one of the above implementations is implemented. A step of.

To sum up, the present invention divides the acquired bird's-eye view image into multiple detection areas, by identifying whether there is a preset shape in each detection area and calculating the proportion of preset color pixels and other color pixel points in each detection area Compared, it can quickly locate the specific position of the text on the item and recognize the text information, with a small amount of calculation, and will not miss any text information that needs to be recognized.

It should be understood that although this description is described according to implementation modes, not each implementation mode only contains an independent technical solution, and this description in the description is only for clarity, and those skilled in the art should take the description as a whole, and each The technical solutions in the embodiments can also be properly combined to form other embodiments that can be understood by those skilled in the art.

The series of detailed descriptions listed above are only specific descriptions for feasible implementations of the present invention, and they are not intended to limit the protection scope of the present invention. Any equivalent implementation or implementation that does not depart from the technical spirit of the present invention All changes should be included within the protection scope of the present invention.

Claims

A method for identifying item information in a refrigerator, comprising the steps of:

Obtain a bird's-eye view image of the space inside the refrigerator bottle holder;

dividing the bird's-eye view image into a plurality of detection areas;

identifying whether there is edge information of a preset shape in the detection area;

If there is edge information of a preset shape, calculating the ratio of the preset color pixel and other color pixels in the detection area;

If the proportion of the pixel of the preset color and the pixel of other colors in the detection area is within the set threshold range, the pixel of the preset color is located, and text information is identified according to the pixel of the preset color.
The method for identifying item information in a refrigerator according to claim 1, wherein "segmenting the bird's-eye view image into multiple detection areas" specifically includes:

The detection area is a square structure;

The detection area is at least larger than the area of the top cover of the item in the bottle holder of the refrigerator;

There is a partial overlap between every two adjacent detection areas.
The method for identifying item information in a refrigerator according to claim 2, wherein "there is a partial overlap between every two adjacent detection areas" is specifically:

There is a partial overlapping area between every two adjacent detection areas, and the overlapping area is half of the detection area.
The method for identifying item information in a refrigerator according to claim 3, wherein "identifying whether there is edge information of a preset shape in the detection area" specifically includes:

Using a circle detection algorithm and/or a square detection algorithm to identify whether there is circular edge information and/or square edge information in the detection area.
The method for identifying item information in a refrigerator according to claim 4, wherein "calculating the ratio of preset color pixels and other color pixels in the detection area" specifically includes:

Acquire all preset color pixels in the detection area;

The preset color is specifically black.
The method for identifying item information in a refrigerator according to claim 5, wherein, "if the ratio of the pixels of the preset color and the pixels of other colors in the detection area is within a set threshold range, then locate the pixel of the preset color." Setting color pixels, and identifying text information according to the preset color pixels" specifically includes:

The threshold range is specifically 30% to 40%;

Use the OCR algorithm to locate the specific position of the preset color pixel, and recognize the text information according to the preset color pixel.
The method for identifying item information in a refrigerator according to claim 6, wherein the method comprises:

Identify whether there is edge information of a preset shape in each detection area one by one.
The method for identifying item information in a refrigerator according to claim 7, wherein the method further comprises:

If there is a preset shape in the detection area, calculate the proportion of the preset color pixel and other color pixels in the detection area;

If the proportion of the preset color pixel and other color pixels in the detection area is within the set threshold range, the preset color pixel is located, and the text information is identified according to the preset color pixel.
The method for identifying item information in a refrigerator according to claim 8, further comprising:

If there is no preset shape in the detection area, continue to identify the next detection area;

If the ratio of preset color pixels and other color pixels in the detection area is not within the set threshold range, continue to identify the next detection area.
A refrigerator, comprising: a bottle holder, a camera, a memory and a processor, characterized in that,

The camera is located at the middle position on the top of the bottle seat and is set vertically downward;

The bottom of the bottle seat is used for placing articles;

The memory stores a computer program that can run on the processor, and the processor implements the steps of the method for identifying item information in the refrigerator according to any one of claims 1-9 when executing the program.