WO2023109151A1

WO2023109151A1 - Method for identifying information of item in refrigerator, and refrigerator

Info

Publication number: WO2023109151A1
Application number: PCT/CN2022/110226
Authority: WO
Inventors: 高洪波; 孔令磊
Original assignee: 青岛海尔电冰箱有限公司; 海尔智家股份有限公司
Priority date: 2021-12-14
Filing date: 2022-08-04
Publication date: 2023-06-22
Also published as: CN114863265A

Abstract

A method for identifying information of an item in a refrigerator, and a refrigerator. The method comprises: obtaining an aerial view image and identifying a contour box in the aerial view image, and performing segmentation in the aerial view image to obtain multiple detection boxes; selecting a contour line, and identifying the orientation of the contour line in the contour box; selecting a detection line, the orientation of the detection line in the detection box being opposite to the orientation of the contour line in the contour box; and according to whether the relative orientation of the detection line and the contour line is the same as the orientation of the contour line in the contour box, determining that a corresponding item is a mirrored item or a real item. The method can eliminate the interference of the mirrored item and improve the accuracy of an identification algorithm.

Description

Method for identifying item information in refrigerator and refrigerator

This application claims the priority of a Chinese patent application with an application date of December 14, 2021, an application number of 202111529525.0, and an invention titled "Method for identifying item information in a refrigerator, refrigerator and computer storage medium", the entire contents of which are incorporated by reference in this application.

technical field

The invention relates to the field of refrigeration devices, in particular to a method for identifying item information in a refrigerator and a refrigerator.

Background technique

With the development of intelligent electrical appliances, the identification of internal items has become a necessary function of smart refrigerators. Usually, in order to realize the identification of the items stored in the refrigerator, one or more cameras are installed inside the refrigerator to collect photos of the items in the refrigerator, and the collected photos are identified and processed and then sent to the user terminal for viewing by the user. Setting the camera above the items can avoid the problem of items occluding each other, and improve the accuracy of item information identification.

However, the items placed inside the bottle holder will produce shadows on the glass cabinet of the bottle holder. The photos of the items collected by the existing camera installed in the middle of the bottle holder often have the problem of item mirroring, and it is easy to identify one item as two items. Seriously affect the recognition effect.

In the existing method, the wall surface of the bottle seat where the items are placed is generally made of frosted material to eliminate the influence of the mirror image of the items, but this method not only increases the production process, but also affects the appearance of the whole machine.

Contents of the invention

The object of the present invention is to provide a method for identifying item information in a refrigerator and a refrigerator.

In order to achieve one of the purposes of the above invention, an embodiment of the present invention provides a method for identifying information on items in a refrigerator, including steps:

Obtain a bird's-eye view image of the space inside the refrigerator bottle holder;

identifying a contour box composed of intersecting contour lines in the bird's eye view image;

Segmenting a plurality of detection frames in the bird's-eye view image, each of which corresponds to an item;

selecting a contour line, and identifying the orientation of the contour line in the contour frame;

Selecting a detection line, the orientation of the detection line in the detection frame is opposite to the orientation of the contour line in the contour frame;

If the relative orientation of the detection line and the contour line is the same as the orientation of the contour line in the contour frame, the corresponding item is a mirror image item;

If the relative orientation of the detection line to the contour line is opposite to the orientation of the contour line in the contour frame, the corresponding item is a real item.

As a further improvement of an embodiment of the present invention, "obtaining a bird's-eye view image of the space in the bottle holder of the refrigerator" specifically includes:

A bird's-eye view image of the space inside the bottle seat is obtained by controlling the camera arranged in the middle of the top of the bottle seat.

As a further improvement of an embodiment of the present invention, "identifying a contour box composed of intersecting contour lines in the bird's-eye view image" specifically includes:

Using an edge detection algorithm, demarcate the contour lines intersecting in the central region of the bird's-eye view image;

The contour lines intersect to form a contour box.

As a further improvement of an embodiment of the present invention, "segmenting a plurality of detection frames in the bird's-eye view image, each of which corresponds to an item" specifically includes:

Use the Yolo V3 target detection algorithm to calibrate the detection frame corresponding to each item in the bird's-eye view image;

The items in the bird's-eye view image include real items placed in the bottle holder and mirrored items corresponding to the real items.

As a further improvement of an embodiment of the present invention, before "selecting a detection line, the orientation of the detection line located in the detection frame is opposite to the orientation of the contour line located in the outline frame" further includes steps:

Obtain the detection frames on both sides adjacent to the contour line.

As a further improvement of an embodiment of the present invention, the method specifically includes:

Select a contour line, the contour line is located at the upper edge of the contour box;

Select a detection line, the detection line is located at the lower line of the detection frame;

If the detection line is above the contour line, the corresponding item is a mirror image item;

If the detection line is located below the contour line, the corresponding item is a real item.

Select a contour line, the contour line is located at the lower edge of the contour box;

Select a detection line, the detection line is located on the upper line of the detection frame;

If the detection line is located above the contour line, the corresponding item is a real item;

If the detection line is located below the contour line, the corresponding item is a mirror image item.

Select a contour line, the contour line is located at the left line of the contour box;

Select a detection line, the detection line is located on the right line of the detection frame;

If the detection line is located to the left of the contour line, the corresponding item is a mirror image item;

If the detection line is located to the right of the contour line, the corresponding item is a real item.

As a further improvement in one embodiment of the present invention, the method also includes the steps of:

Exclude the mirrored items, and count the number of real items in the bird's-eye view image.

An embodiment of the present invention also provides a refrigerator, including: a bottle holder, a camera, a memory, and a processor, wherein,

The bottle holder includes a bottle holder top, a bottom and a bottle holder wall, and the camera is arranged at the middle position of the bottle holder top and is arranged vertically downward;

The bottom of the bottle seat is used for placing articles;

At least one side of the bottle seat wall is a glass surface;

The memory stores a computer program that can run on the processor, and the processor implements the steps of the method for identifying item information in the refrigerator as described in any one of the above implementation manners when executing the program.

Compared with the prior art, the present invention adopts the edge detection algorithm and the Yolo V3 target detection algorithm to calibrate the contour frame and the detection frame corresponding to the item, obtain the contour line and the detection line, the detection line is located in the direction of the detection frame and the contour line is located in the contour frame The orientation is opposite. By comparing whether the relative orientation of the detection line and the contour line is the same as the orientation of the contour line in the outline frame, it can be judged whether the object detected in the bird's-eye view image is a real object or a mirror image, so as to eliminate the interference of the mirror image and improve recognition. Algorithm accuracy.

Description of drawings

Fig. 1 is a schematic flowchart of a method for identifying information on items in a refrigerator in an embodiment of the present invention.

Fig. 2(a) is a schematic diagram of the space inside the refrigerator bottle holder in Embodiment 1 of the present invention (except for the outline for illustration and the items on adjacent sides, the rest of the content in the figure is omitted).

FIG. 2( b ) is a schematic flow chart of a method for identifying item information in a refrigerator in Embodiment 1 of the present invention.

Fig. 3(a) is a schematic diagram of the space inside the bottle holder of the refrigerator in the second embodiment of the present invention (except for the outline for illustration and the items on the adjacent sides, the rest of the content in the figure is omitted).

Fig. 3(b) is a schematic flowchart of a method for identifying item information in a refrigerator in Embodiment 2 of the present invention.

Fig. 4(a) is a schematic diagram of the space inside the bottle holder of the refrigerator in the third embodiment of the present invention (except for the outline for illustration and the items on the adjacent sides, the rest of the content in the figure is omitted).

FIG. 4( b ) is a schematic flow chart of a method for identifying item information in a refrigerator in Embodiment 3 of the present invention.

Fig. 5(a) is a schematic diagram of the space inside the refrigerator bottle holder in Embodiment 4 of the present invention (except for the outline for illustration and the items on adjacent sides, the rest of the content in the figure is omitted).

FIG. 5( b ) is a schematic flowchart of a method for identifying item information in a refrigerator according to Embodiment 4 of the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the following will clearly and completely describe the technical solution of the present application in combination with specific implementation methods of the present application and corresponding drawings. Apparently, the described implementations are only some of the implementations of this application, not all of them. Based on the implementation manners in this application, all other implementation manners obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

For the convenience of description, terms representing relative positions in space are used herein for description, such as "upper", "lower", "rear", "front", etc., which are used to describe the relative position of one unit or feature shown in the drawings relative to another. A unit or feature relationship. Spatially relative terms may encompass different orientations of the device in use or operation other than the orientation shown in the figures. For example, if the device in the figures is turned over, elements described as "below" or "above" other elements or features would then be oriented "below" or "above" the other elements or features. Thus, the exemplary term "below" can encompass both a spatial orientation of below and above.

As shown in Figure 1, the present invention provides a method for identifying item information in a refrigerator, comprising steps:

S1: Obtain a bird's-eye view image of the space inside the refrigerator bottle holder.

In this embodiment, the camera is set on the top of the bottle seat, and the object is photographed from above to obtain a bird's-eye view image. In order to avoid mutual occlusion between the items in the bottle seat, which is not conducive to the detection and identification of item information, the present invention preferably uses the camera It is located in the middle of the top of the bottle seat.

Certainly, in some other embodiments of the present invention, two or more cameras may also be arranged in the bottle holder to acquire two or more bird's-eye-view images of the space inside the refrigerator bottle holder taken from different angles.

S2: Identify a contour box composed of intersecting contour lines in the bird's-eye view image.

Because the bottom of the bottle base and the wall of the bottle base are made of different materials, the areas of the bottom of the bottle base and the wall of the bottle base captured by the camera usually have obvious pixel grayscale differences in the image, and obvious edge contours can be detected and identified.

Specifically, an embodiment of the present invention utilizes an edge detection algorithm to identify the contour line of the edge of the bottle seat, and marks the contour lines intersecting in the middle region of the bird's-eye view image, and the contour lines intersect to form a contour frame 1, as shown in Figure 2(a)- 5(a). Here, the edge detection algorithm can adopt currently common algorithms such as Roberts, Sobel, Prewitt, Canny, Log, etc., and the specific algorithm content is the prior art, and will not be repeated here.

S3: Segment multiple detection frames in the bird's-eye view image, each of which corresponds to an item.

When the wall of the bottle seat does not use frosted material, the items in the bottle seat will often cast corresponding shadows on the wall of the bottle seat, that is, its mirror image. The real item in the bottle holder and the mirror image item corresponding to the real item.

Specifically, identify the items in the bird's-eye view image, and use the Yolo V3 target detection algorithm to calibrate the detection frame 2 corresponding to each item in the bird's-eye view image. The specific content of the Yolo V3 target detection algorithm here is the existing technology, so I will not repeat it here.

S4: Select a contour line, and identify the position where the contour line is located in the contour box.

Specifically, the outline frame 1 is a square structure, which includes a contour line 11 positioned above the contour frame, a contour line 12 positioned below the contour frame, a contour line 13 positioned on the left side of the contour frame, and a contour line positioned on the right side of the contour frame. Azimuth of the contour line 14 .

Exemplarily, in the implementation manner of the present invention, selecting a contour line may be contour line 11 , contour line 12 , contour line 13 or contour line 14 .

S5: Select a detection line, the orientation of the detection line in the detection frame is opposite to the orientation of the contour line in the outline frame.

In one embodiment of the present invention, the detection frame 2 is a square structure, which includes a detection line 21 located above the detection frame, a detection line 22 located below the detection frame, a detection line 23 located on the left side of the detection frame, And the detection line 24 located on the right side of the detection frame.

Here, the selected detection line may be the detection line 21 , the detection line 22 , the detection line 23 or the detection line 24 . In the embodiment of the present invention, the orientation of the selected detection line in the detection frame 2 is opposite to the orientation of the contour line in the contour frame 1. Exemplarily, if the contour line 11 is selected in step S4, then the detection line is selected in step S5. Line 22; If choose contour line 12 in step S4, then choose detection line 21 in step S5; If choose contour line 13 in step S4, then choose detection line 24 in step S5; If choose contour line 14 in step S4 , then the detection line 23 is selected in step S5.

Furthermore, before "selecting a detection line", a step is also included: acquiring a detection frame adjacent to the contour line. Exemplarily, if the contour line 11 is selected in step S4, the system only acquires the object detection frames existing on two surfaces adjacent to the contour line 11, and does not use this as a judgment standard for object detection frames on other surfaces.

According to whether the relative orientation of the selected detection line and the contour line is the same as the orientation of the contour line in the outline frame 1, it is judged whether the item identified in the bird's-eye view image is a real item or a mirror image item.

S61: If the relative orientation of the detection line and the contour line is the same as the orientation of the contour line in the outline frame 1, then the corresponding item is a mirror image item.

S62: If the relative orientation of the detection line to the contour line is opposite to the orientation of the contour line in outline frame 1, the corresponding item is a real item.

Exemplarily, as shown in Figure 2(a), for an item a identified in a bird's-eye view image, its detection frame 2a includes 21a located in the upper position, 22a located in the lower position, 23a located in the left position and 24a located in the right position . If the contour line 11 is selected in step S4, then the detection line 22a is selected as the judgment standard in the embodiment of the present invention in the step S5, because the relative orientation of the detection line 22a and the contour line 11 is that the detection line 22a is located at the contour line 11, and the contour line 11 is located at the upper position of the contour frame 1, so for the item a, the relative orientation of the detection line 22a and the contour line 11 is the same as that of the contour line 11 in the contour frame 1, that is, step S61 is executed, and the system Determine that item a is a mirror image item.

For the item b identified in the bird's-eye view image, its detection frame 2b includes 21b at the upper position, 22b at the lower position, 23b at the left position and 24b at the right position. If the contour line 11 is selected in step S4, then the detection line 22b is selected as the judgment standard in the embodiment of the present invention in the step S5, because the relative orientation of the detection line 22b and the contour line 11 is that the detection line 22b is located at the contour line 11 , and the contour line 11 is located at the upper position of the contour frame 1, so for the item b, the relative orientation between the detection line 22b and the contour line 11 is opposite to the orientation of the contour line 11 in the contour frame 1, that is, step S62 is executed to determine the object b is the real item.

For item c and item d in Figure 2(a), according to the above judgment method, it can be judged that item c is a mirror image item, and item d is a real item.

Since the selected contour lines and detection lines are different, the present invention provides four specific methods for identifying items in refrigerator bottle holders, which are described in detail in the following four embodiments.

Embodiment one

In embodiment one, select contour line 11, as shown in Figure 2 (a), described identification method (Figure 2 (b)) comprises steps:

S4A1: Select a contour line 11 , the contour line 11 is located on the upper edge of the contour frame 1 .

S5A1: Select a detection line 22 , the detection line 22 is located at the lower line of the detection frame 2 .

S61A1: If the detection line 22 is located above the contour line 11, the corresponding item is a mirror image item.

S62A1: If the detection line 22 is located below the contour line 11, the corresponding item is a real item.

Similarly, before selecting a detection line 22 , a step is further included: obtaining the detection frames on both sides adjacent to the contour line 11 . As shown in Figure 2(a), for the item a in the figure, since the detection line 22a is above the contour line 11, it is judged that the object a is a mirror image; for the item b in the figure, since the detection line 22b is located below the contour line 11, it is judged that the object b is a real item; for the item c in the figure, since the detection line 22c is located above the contour line 11, it is judged that the item c is a mirror image item; for the item d in the figure, since the item d is a large item with a high height, such as a large bottle In the acquired bird's-eye view, the detection frame 2d corresponding to the item d straddles the adjacent two sides of the contour line 11, but since the detection line 22d is located below the contour line 11, the object can also be judged according to the above method d is the real item. The following other three methods for judging real items or mirrored items in the implementation can all be used to identify large items in the bottle holder, and there will be no difference in the recognition effect due to the different volumes of the items.

In this embodiment, the object detection frames on both sides of the contour line 11 acquired in the bird's-eye view can use the object information identification method in FIG. 2( b ) to determine whether the object is a real object or a mirror image.

Embodiment two

In embodiment two, choose contour line 12, as shown in Figure 3 (a), described identification method (Figure 3 (b) comprises steps:

S4B1: Select a contour line 12, the contour line 12 is located at the lower edge of the contour box 1.

S5B1: Select a detection line 21 , the detection line 21 is located on the upper line of the detection frame 2 .

S61B1: If the detection line 21 is located above the contour line 12, the corresponding item is a real item.

S62B1: If the detection line 21 is located below the contour line 12, the corresponding item is a mirror image item.

Similarly, before selecting a detection line 21, a step is also included: obtaining the detection frames on both sides adjacent to the contour line 12. As shown in Figure 3(a), for the item e in the figure, since the detection line 21e is located above the contour line 12, it is judged that the item e is a real item; for the item f in the figure, since the detection line 21f is located below the contour line 12, it is judged that Item f is a mirrored item.

In this embodiment, the object detection frames on both sides of the contour line 12 obtained in the bird's-eye view can use the object information identification method in FIG. 3( b ) to determine whether the object is a real object or a mirror image.

Embodiment three

In embodiment three, choose contour line 13, as shown in Figure 4 (a), described identification method (Figure 4 (b) comprises steps:

S4C1: Select a contour line 13, the contour line 13 is located at the left line of the contour box 1.

S5C1: Select a detection line 24 , the detection line 24 is located on the right line of the detection frame 2 .

S61C1: If the detection line 24 is located to the left of the contour line 13, the corresponding item is a mirror image item.

S62C1: If the detection line 24 is located to the right of the contour line 13, the corresponding item is a real item.

Similarly, before selecting a detection line 24 , a step is further included: obtaining the detection frames on both sides adjacent to the contour line 13 . As shown in Figure 4(a), for the item g in the figure, since the detection line 24g is located on the left side of the contour line 13, it is judged that the item g is a mirror image item; for the item h in the figure, since the detection line 24h is located on the right side of the contour line 13 , that is, it is judged that the item h is a real item.

In this embodiment, the object detection frames on both sides of the contour line 13 obtained in the bird's-eye view can use the object information identification method in FIG. 4( b ) to determine whether the object is a real object or a mirror image.

Embodiment four

In embodiment four, choose contour line 14, as shown in Figure 5 (a), described identification method (Figure 5 (b) comprises steps:

S4D1: Select a contour line 14 , the contour line 14 is located on the right line of the contour box 1 .

S5D1: Select a detection line 23 , the detection line 23 is located on the left line of the detection frame 2 .

S61D1: If the detection line 23 is located to the left of the contour line 14, the corresponding item is a real item.

S62D1: If the detection line 23 is located to the right of the contour line 14, the corresponding item is a mirror image item.

Similarly, before selecting a detection line 23 , a step is further included: obtaining the detection frames on both sides adjacent to the contour line 14 . As shown in Fig. 5 (a), for the item i in the figure, since the detection line 23i is located on the left side of the contour line 14, it is judged that the item i is a real item; for the item j in the figure, since the detection line 23j is located on the contour line On the right side of 14, it is judged that item j is a mirror image item.

In this embodiment, the object detection frames on both sides of the contour line 14 acquired in the bird's-eye view can use the object information identification method in FIG. 5( b ) to determine whether the object is a real object or a mirror image.

Further, the method further includes steps after identifying the item in the bird's-eye view as a real item or a mirror image item:

S7: Exclude mirrored items, and count the number of real items in the bird's-eye view image.

The present invention also provides a refrigerator, which includes a bottle holder, a camera, a memory and a processor.

Further, the bottle seat includes a top, a bottom, and a wall of the bottle seat, and the camera is set at the middle of the top of the bottle seat and is set vertically downward.

The bottom of the bottle seat is used for placing articles, and at least one side of the wall of the bottle seat is a glass surface.

The memory stores a computer program that can run on the processor, and when the processor executes the program, the steps of the above-mentioned method for identifying item information in the refrigerator are realized.

In summary, the present invention uses the edge detection algorithm and the Yolo V3 target detection algorithm to calibrate the contour frame and the detection frame corresponding to the item, and obtain the contour line and the detection line. , by comparing whether the relative orientation of the detection line and the contour line is the same as the orientation of the contour line in the contour box, it is judged whether the object detected in the bird's-eye view image is a real object or a mirror image, so as to eliminate the interference of the mirror image and improve the recognition algorithm. Accuracy.

It should be understood that although this description is described according to implementation modes, not each implementation mode only includes an independent technical solution. The technical solutions in the embodiments can also be properly combined to form other embodiments that can be understood by those skilled in the art.

The series of detailed descriptions listed above are only specific descriptions of the feasible implementation modes of the present invention, and they are not intended to limit the protection scope of the present invention. All changes should be included within the protection scope of the present invention.

Claims

A method for identifying item information in a refrigerator, comprising the steps of:

Obtain a bird's-eye view image of the space inside the refrigerator bottle holder;

identifying a contour box composed of intersecting contour lines in the bird's eye view image;

Segmenting a plurality of detection frames in the bird's-eye view image, each of which corresponds to an item;

selecting a contour line, and identifying the orientation of the contour line in the contour box;

Selecting a detection line, the orientation of the detection line in the detection frame is opposite to the orientation of the contour line in the contour frame;

If the relative orientation of the detection line and the contour line is the same as the orientation of the contour line in the contour frame, the corresponding item is a mirror image item;

If the relative orientation of the detection line to the contour line is opposite to the orientation of the contour line in the contour frame, the corresponding item is a real item.
The method for identifying item information in a refrigerator according to claim 1, wherein "obtaining a bird's-eye view image of the space in the bottle holder of the refrigerator" specifically includes:

A bird's-eye view image of the space inside the bottle seat is obtained by controlling the camera arranged in the middle of the top of the bottle seat.
The method for identifying item information in a refrigerator according to claim 2, wherein "identifying a contour frame composed of intersecting contour lines in the bird's-eye view image" specifically includes:

Using an edge detection algorithm, demarcate the contour lines intersecting in the central region of the bird's-eye view image;

The contour lines intersect to form a contour box.
The method for identifying item information in a refrigerator according to claim 3, wherein "segmenting a plurality of detection frames in the bird's-eye view image, each of which corresponds to an item" specifically includes:

Use the Yolo V3 target detection algorithm to calibrate the detection frame corresponding to each item in the bird's-eye view image;

The items in the bird's-eye view image include real items placed in the bottle holder and mirrored items corresponding to the real items.
The method for identifying item information in a refrigerator according to claim 4, further comprising a step before "selecting a detection line, the orientation of the detection line being located in the detection frame is opposite to the orientation of the contour line being located in the outline frame" :

Obtain the detection frames on both sides adjacent to the contour line.
The method for identifying item information in a refrigerator according to claim 5, wherein the method specifically comprises:

Select a contour line, the contour line is located at the upper edge of the contour box;

Select a detection line, the detection line is located at the lower line of the detection frame;

If the detection line is above the contour line, the corresponding item is a mirror image item;

If the detection line is located below the contour line, the corresponding item is a real item.
The method for identifying item information in a refrigerator according to claim 5, wherein the method specifically comprises:

Select a contour line, the contour line is located at the lower edge of the contour box;

Select a detection line, the detection line is located on the upper line of the detection frame;

If the detection line is located above the contour line, the corresponding item is a real item;

If the detection line is located below the contour line, the corresponding item is a mirror image item.
The method for identifying item information in a refrigerator according to claim 5, wherein the method specifically comprises:

Select a contour line, the contour line is located at the left line of the contour box;

Select a detection line, the detection line is located on the right line of the detection frame;

If the detection line is located to the left of the contour line, the corresponding item is a mirror image item;

If the detection line is located to the right of the contour line, the corresponding item is a real item.
The method for identifying item information in a refrigerator according to claim 1, further comprising the steps of:

Exclude the mirrored items, and count the number of real items in the bird's-eye view image.
A refrigerator, comprising: a bottle holder, a camera, a memory and a processor, characterized in that,

The bottle holder includes a bottle holder top, a bottom and a bottle holder wall, and the camera is arranged at the middle position of the bottle holder top and is arranged vertically downward;

The bottom of the bottle seat is used for placing articles;

At least one side of the bottle seat wall is a glass surface;

The memory stores a computer program that can run on the processor, and when the processor executes the program, the steps of the method for identifying item information in the refrigerator according to any one of claims 1-9 are realized.