CN110705371B - Refrigerator structure and commodity surface arrangement detection method thereof - Google Patents
Refrigerator structure and commodity surface arrangement detection method thereof Download PDFInfo
- Publication number
- CN110705371B CN110705371B CN201910848229.3A CN201910848229A CN110705371B CN 110705371 B CN110705371 B CN 110705371B CN 201910848229 A CN201910848229 A CN 201910848229A CN 110705371 B CN110705371 B CN 110705371B
- Authority
- CN
- China
- Prior art keywords
- ice chest
- door
- ice
- refrigerator
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Quality & Reliability (AREA)
- Multimedia (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)
Abstract
The invention discloses a refrigerator structure and a commodity surface arrangement detection method thereof, which relate to the field of image recognition methods of commodities in commercial refrigerators and comprise a model training module and a detection implementation module, wherein the detection implementation module comprises the following steps: (1) starting; (2) detecting the refrigerator door in the original drawing; (3) cropping the ice bin door map from the artwork; (4) detecting an ice bin level on the ice bin door map; (5) cutting each of said ice chest layers to form an ice chest layer map; (6) detecting and identifying merchandise on the ice bin map; (7) restoring the position of the commodity in the artwork; and (8) ending. The invention can remove the influence of other ice boxes outside the ice boxes or the commodities on the goods shelves or the commodities in the glass reflection on the commodity arrangement surface detection, and simultaneously improves the detection rate of empty layers or non-empty layers.
Description
Technical Field
The invention relates to the field of image recognition methods of commodities in commercial ice chests, in particular to an ice chest structure and a commodity surface arrangement detection method thereof.
Background
In the quick-wear industry, the display of merchandise in retail outlets is directly related to its sales volume, so brands are strongly in need of digitalization of merchandise display, and especially beverage brands want to know the display of merchandise in their own or customer's ice boxes. The traditional practice is that brands send people to the retail channel to manually count the number of lines or to upload the number of lines to the back end through mobile phone photographing before manually counting the number of lines.
In recent years, image recognition algorithms based on deep learning (mainly the faster-rcnn and its derivative algorithms) have begun to play a role in merchandise display digitization. The method comprises the steps of pre-collecting photos, manually labeling the commodities in the photos (labeling the category of each commodity and the binding box thereof) to prepare training data, and training a model based on a master-rcnn algorithm, wherein the obtained model can directly identify the commodity category and the binding box in the photos, and the algorithm accuracy can reach more than 90%. The bounding box of an object refers to the smallest bounding rectangle of the object, defined by the upper left corner (x 1, y 1) and lower right corner (x 2, y 2) of the rectangle.
In the refrigerator-freezer scene, in order to be able to clearly shoot commodity in the refrigerator-freezer, avoid the influence of aqueous vapor and glass reflection of light, often can require the shooting personnel to pull open the refrigerator door and shoot, and the glass on the door that pulls open often can form bilateral symmetry's reflection to commodity in the refrigerator-freezer, commodity in this reflection can be discerned by the model. Meanwhile, in order to avoid missing the commodity in the refrigerator in the shooting process, the shooting range is often enlarged, and other refrigerator or goods shelf scenes beside the target refrigerator are common, so that the commodity outside the refrigerator or in the reflection can be counted after identification, and the commodity display has larger deviation with the actual commodity display.
Additionally, brands also want to know the number of layers of the target ice bin to calculate the saturation and absence indicators. The existing method can detect the layer through the horizontal line segment of the layer, but the actual scene often has low detection rate due to the discontinuous horizontal line segment caused by factors such as price tags or promotion materials placed on the layer; the number of floors can be estimated by the display of the merchandise itself, but cannot be estimated for empty floors without merchandise placement.
Accordingly, those skilled in the art have been working to develop an ice bin structure and method for detecting the level of merchandise in a target ice bin.
Disclosure of Invention
In view of the above-mentioned drawbacks of the prior art, the present invention aims to solve the technical problem of being able to remove the influence of other ice boxes outside the ice box or the commodities on the shelf or the commodities in the glass reflection on the commodity surface detection, and simultaneously improve the detection rate of empty or non-empty layers.
In order to achieve the above purpose, the invention provides a refrigerator structure and a commodity surface arrangement detection method thereof, which is characterized by comprising a model training module and a detection implementation module; wherein the detection implementation module is implemented based on the model training module and comprises the following steps:
step 1, starting;
step 2, detecting a refrigerator door in an original image, wherein the detection result is a binding box of the refrigerator door in the original image;
step 3, cutting the refrigerator door from the original image according to the binding box in the step 2 to form a refrigerator door diagram;
step 4, detecting the ice chest layers on the ice chest door diagram, wherein the detection result is that each ice chest layer corresponds to a binding box of the ice chest door diagram;
step 5, cutting each ice chest layer from the ice chest door map according to the binding box in the step 4 to form an ice chest layer map;
step 6, detecting and identifying commodities on the ice chest layer diagram, wherein the detection result is a binding box of each commodity relative to the ice chest layer diagram, and the identification result is the specific classification of each commodity and the confidence coefficient thereof;
step 7, according to the relative coordinate position of each ice chest layer in the ice chest door diagram and the relative coordinate position of the ice chest door in the original diagram, reducing the boundin box of the commodity back to the original diagram to obtain the boundin box in the original diagram;
and 8, ending.
Further, the model training module comprises the following steps:
step 1, collecting original pictures of an ice chest, and manufacturing an ice chest door data set;
step 2, training a refrigerator door detector by using a target detection algorithm based on deep learning;
step 3, cutting the refrigerator door from the original image of the refrigerator according to the marking data, and storing the refrigerator door to manufacture a refrigerator layer data set;
and 4, training the refrigerator layer detector by using a target detection algorithm based on deep learning.
Further, in the step 2 of the detection implementation module, if the refrigerator door is in a closed state, direct detection is performed; and if the refrigerator door is in an open state, detecting the cavity of the refrigerator.
Further, the ice chest layer of the detection implementation module includes an empty layer and a layer with merchandise placement.
Further, wherein the ice bin layer of the detection implementation module refers to the artwork inner barrier layer itself and the space above where merchandise may be placed.
Further, the step 1 of the model training module creates the data set of the door of the ice chest to ensure that the number of the door of the ice chest in the closed state is as large as possible as the number of the door of the ice chest in the open state.
Further, the step 3 of the model training module further includes selecting the original ice chest with more empty layers for detection.
Further, the step 3 of the detection implementation module further comprises saving the ice chest door map.
Further, the step 5 of the detection implementation module further comprises saving the ice bin layer map.
The invention also provides an alternative method of the refrigerator structure and the commodity surface arrangement detection method thereof, which is characterized by comprising the following steps:
step 1, starting;
step 2, detecting the refrigerator door, the refrigerator layer and the commodity in original pictures;
step 3, removing the ice chest layer and the commodity outside the ice chest door;
and step 4, ending.
The invention has the following remarkable technical effects:
1. the refrigerator door is detected, so that interference information outside a target area is removed, and the statistical result is more accurate;
2. the detection rate is improved by detecting the refrigerator layer (the interlayer and the upper space);
3. and commodity identification is carried out on the cut picture, so that the operation of shrinking the original picture is avoided, and the detection rate and the accuracy are improved.
The conception, specific structure, and technical effects of the present invention will be further described with reference to the accompanying drawings to fully understand the objects, features, and effects of the present invention.
Drawings
FIG. 1 is a flow chart of a preferred embodiment of the present invention;
fig. 2 is a flow chart of another preferred embodiment of the present invention.
Detailed Description
The following description of the preferred embodiments of the present invention refers to the accompanying drawings, which make the technical contents thereof more clear and easy to understand. The present invention may be embodied in many different forms of embodiments and the scope of the present invention is not limited to only the embodiments described herein.
Example 1
A flow chart of a preferred embodiment of the present invention is shown in fig. 1.
Before the detection is implemented, the deep learning-based method is firstly used for training the refrigerator door detector and the refrigerator layer detector, and the specific operation steps are as follows:
1. collecting original pictures of various scenes, and ensuring that the number of photos with doors being opened is equal to that of photos with doors being closed as much as possible, for example, 500 photos with doors being opened and 500 photos with doors being closed.
2. Making a refrigerator door data set:
a. for a door-opening refrigerator, marking a refrigerator cavity so that a binding box just covers the refrigerator;
b. for a closed ice chest, the ice chest door is such that the binding box covers exactly.
3. The ice chest door detector is trained with a deep learning based target detection algorithm (e.g., faster RCNN and its derivative algorithms).
4. And cutting the refrigerator door from the original drawing according to the marking data, and storing the refrigerator door for manufacturing the refrigerator layer data set.
5. If the number of samples containing empty layers is small, in order to keep sample balance, the detection rate of the empty layers is improved, and then 100 pictures containing more empty layers are selected and added into the data set according to the methods of 2 and 4.
6. Making a freezer layer dataset: the ice chest layer is marked so that the binding box just covers.
7. The ice chest level detector is trained with a deep learning based target detection algorithm.
8. The detection and identification of the ice chest structure and the products therein is performed after the ice chest door detector, ice chest layer detector, and product detection and identification (assuming present, not within this patent range) are obtained.
The specific operation of detecting and identifying the refrigerator structure and the goods in the refrigerator structure based on the training model is carried out according to the following steps:
1. starting.
2. Detecting the door of the refrigerator on the original image, and directly detecting if the refrigerator is in a door closing state; and if the refrigerator is in a door opening state, detecting the cavity of the refrigerator. The detection result is a binding box with the refrigerator door in the original image. The refrigerator door can remove commodities and refrigerator layers outside the target area (including commodities in a side refrigerator or a goods shelf and commodities in glass reflection after the door is opened) after detection, interference items are shielded, and the detection rate can reach 99%.
3. Cutting the refrigerator door from the original image according to the detected binding box;
4. and detecting an ice chest layer on the cut ice chest door diagram, wherein the ice chest layer comprises an empty layer and a layer with goods placed. The test results are the binding boxes of the ice chest door diagrams for each ice chest layer. This step enables detection of ice chest layers, including empty layers and non-empty layers. The refrigerator layer refers to the inner interlayer of the refrigerator and a space above which commodities can be placed, and the detection rate can reach 99%. The number of layers required in service calculation can be directly obtained according to the detection result of the ice chest layer.
5. Cutting each ice chest layer from the ice chest door map according to the detected binding boxes;
6. the merchandise is detected and identified on the cut ice bin floor map. The detection result is a binding box of each commodity relative to the ice chest layer diagram, and the identification result is the specific classification of each commodity and the confidence level thereof. Detecting the merchandise on the cut-down bin floor map corresponds to performing an enlargement operation on the merchandise area compared to detecting the merchandise directly on the original map. Generally, a picture to be identified, such as 1080p (1920 x 1080), is limited by a video memory, and is subjected to reduction processing before identification, and commodity characteristic information is partially lost in the original image reduction process. However, the cutting operation is performed on the original image, the cut image can be directly identified without being reduced, more characteristic details can be reserved, and the detection rate (from 98% to 99%) and the identification accuracy (from 95% to 96%) are improved.
7. And restoring the bound box of the commodity to the original image according to the relative coordinate position of each layer in the refrigerator door image and the coordinate position of the refrigerator door in the original image to obtain the binding box in the original image.
8. And (5) ending.
Example two
Fig. 2 is a flow chart of another preferred embodiment of the present invention. In the embodiment, the detection of the refrigerator door, the detection of the refrigerator layer and the detection and identification of the goods are all carried out on original pictures, and then the goods outside the door and the refrigerator layer are removed. The alternative method provided by this embodiment differs from the method provided by the first embodiment in that both detection and identification are performed on the original image, the identification is not cut, and the masking and rejecting processes are added in the later process. Compared with the original method, the method only carries out three times of detection (such as a four-layer single-door refrigerator, n=1, m=4, and 6 times of detection) on the original image, but the speed is faster, the characteristic detail loss is more because of no cutting and amplifying process, and the detection rate and the recognition accuracy are reduced compared with the original method.
The invention uses a deep learning method to take the refrigerator door (door opening or door closing) as a detection target, and eliminates the interference information outside the target area; the deep learning method is used for taking the refrigerator layer (the interlayer and the space above the interlayer) as a detection target, so that the problem that the interlayer is difficult to detect through line segments is solved; the detection of the refrigerator layer and the commodity is carried out on the cut graph, which is equivalent to the amplification operation of the target area, more characteristic details are reserved, and the detection rate and the identification are improved.
The foregoing describes in detail preferred embodiments of the present invention. It should be understood that numerous modifications and variations can be made in accordance with the concepts of the invention without requiring creative effort by one of ordinary skill in the art. Therefore, all technical solutions which can be obtained by logic analysis, reasoning or limited experiments based on the prior art by the person skilled in the art according to the inventive concept shall be within the scope of protection defined by the claims.
Claims (8)
1. The refrigerator structure and the commodity surface arrangement detection method thereof are characterized by comprising a model training module and a detection implementation module; wherein the detection implementation module is implemented based on the model training module and comprises the following steps:
step 1, starting;
step 2, detecting a refrigerator door in an original image, wherein the detection result is a binding box of the refrigerator door in the original image;
step 3, cutting the refrigerator door from the original image according to the binding box in the step 2 to form a refrigerator door diagram;
step 4, detecting the ice chest layers on the ice chest door diagram, wherein the detection result is that each ice chest layer corresponds to a binding box of the ice chest door diagram;
step 5, cutting each ice chest layer from the ice chest door map according to the binding box in the step 4 to form an ice chest layer map;
step 6, detecting and identifying commodities on the ice chest layer diagram, wherein the detection result is a binding box of each commodity relative to the ice chest layer diagram, and the identification result is the specific classification of each commodity and the confidence coefficient thereof;
step 7, according to the relative coordinate position of each ice chest layer in the ice chest door diagram and the relative coordinate position of the ice chest door in the original diagram, restoring the commodity binding box back to the original diagram,
obtaining a binding box in the original image;
step 8, ending;
wherein, the model training module comprises the following steps:
step 1, collecting original pictures of an ice chest, and manufacturing an ice chest door data set;
step 2, training a refrigerator door detector by using a target detection algorithm based on deep learning;
step 3, cutting the refrigerator door from the original image of the refrigerator according to the marking data, and storing the refrigerator door to manufacture a refrigerator layer data set;
and 4, training the refrigerator layer detector by using a target detection algorithm based on deep learning.
2. The ice chest structure and method for detecting a commodity surface according to claim 1, wherein in said step 2 of said detection implementation module, if said ice chest door is in a closed state, it is directly detected; and if the refrigerator door is in an open state, detecting the cavity of the refrigerator.
3. The ice bin structure of claim 1, and a method of detecting a level of merchandise in said ice bin structure, wherein said ice bin layer of said detection implementation module comprises an empty layer and a layer having merchandise disposed thereon.
4. The ice chest structure and method of detecting a commodity surface according to claim 1, wherein said ice chest layer of said detection implementation module is the artwork inner barrier itself and a space above which a commodity can be placed.
5. The ice chest structure and method for detecting a commodity surface according to claim 1, wherein said step 1 of said model training module creates said ice chest door dataset to ensure as much as possible that said ice chest doors in a closed condition are consistent with said ice chest doors in an open condition.
6. The ice chest structure and method for detecting a commodity surface according to claim 1, wherein said step 3 of said model training module further comprises selecting said ice chest artwork having a greater number of empty layers for detection.
7. The ice bin structure of claim 1, and a method of detecting a commodity level thereof, wherein said step 3 of said detection implementation module further comprises saving said ice bin door map.
8. The ice bin structure and method of detecting a commodity level according to claim 1, wherein said step 5 of said detection implementation module further comprises saving said ice bin floor map.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910848229.3A CN110705371B (en) | 2019-09-09 | 2019-09-09 | Refrigerator structure and commodity surface arrangement detection method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910848229.3A CN110705371B (en) | 2019-09-09 | 2019-09-09 | Refrigerator structure and commodity surface arrangement detection method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110705371A CN110705371A (en) | 2020-01-17 |
| CN110705371B true CN110705371B (en) | 2023-06-06 |
Family
ID=69194869
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910848229.3A Active CN110705371B (en) | 2019-09-09 | 2019-09-09 | Refrigerator structure and commodity surface arrangement detection method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110705371B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111681234A (en) * | 2020-06-11 | 2020-09-18 | 名创优品（横琴）企业管理有限公司 | Method, system and equipment for detecting standard of trial product placed on store shelf |
| CN112699778A (en) * | 2020-12-29 | 2021-04-23 | 上海零眸智能科技有限公司 | Deep learning-based refrigerator inventory condition supervision and identification method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2015210651A (en) * | 2014-04-25 | 2015-11-24 | サントリーシステムテクノロジー株式会社 | Merchandise identification system |
| CN108596187A (en) * | 2018-03-30 | 2018-09-28 | 青岛海尔智能技术研发有限公司 | Commodity degree of purity detection method and showcase |
| CN108960318A (en) * | 2018-06-28 | 2018-12-07 | 武汉市哈哈便利科技有限公司 | A kind of commodity recognizer using binocular vision technology for self-service cabinet |
| CN109543527A (en) * | 2018-10-19 | 2019-03-29 | 北京陌上花科技有限公司 | For the commodity detection method of unmanned shelf, device and retail terminal |
| CN109872448A (en) * | 2019-01-19 | 2019-06-11 | 创新奇智(广州)科技有限公司 | A kind of foreign matter detecting method, computer readable storage medium and detection system |
-
2019
- 2019-09-09 CN CN201910848229.3A patent/CN110705371B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2015210651A (en) * | 2014-04-25 | 2015-11-24 | サントリーシステムテクノロジー株式会社 | Merchandise identification system |
| CN108596187A (en) * | 2018-03-30 | 2018-09-28 | 青岛海尔智能技术研发有限公司 | Commodity degree of purity detection method and showcase |
| CN108960318A (en) * | 2018-06-28 | 2018-12-07 | 武汉市哈哈便利科技有限公司 | A kind of commodity recognizer using binocular vision technology for self-service cabinet |
| CN109543527A (en) * | 2018-10-19 | 2019-03-29 | 北京陌上花科技有限公司 | For the commodity detection method of unmanned shelf, device and retail terminal |
| CN109872448A (en) * | 2019-01-19 | 2019-06-11 | 创新奇智(广州)科技有限公司 | A kind of foreign matter detecting method, computer readable storage medium and detection system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110705371A (en) | 2020-01-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109214751B (en) | Intelligent inventory management system based on inventory position change | |
| US9697429B2 (en) | Method and apparatus for image processing to avoid counting shelf edge promotional labels when counting product labels | |
| JP6938116B2 (en) | Inventory management device and inventory management method | |
| CN111415461B (en) | Article identification method and system and electronic equipment | |
| CN110705371B (en) | Refrigerator structure and commodity surface arrangement detection method thereof | |
| CN108345912A (en) | Commodity rapid settlement system based on RGBD information and deep learning | |
| CN108596187B (en) | Commodity purity detection method and display cabinet | |
| US10354395B2 (en) | Methods and apparatus to improve detection and false alarm rate over image segmentation | |
| CN113468914B (en) | Method, device and equipment for determining purity of commodity | |
| KR102017980B1 (en) | Refrigerator with displaying image by identifying goods using artificial intelligence and method of displaying thereof | |
| CN112184751A (en) | Object identification method and system and electronic equipment | |
| CN116452636A (en) | Target tracking-based dynamic commodity identification method and related device for unmanned sales counter | |
| CN113128463B (en) | Image recognition method and system | |
| WO2021091481A1 (en) | System for object identification and content quantity estimation through use of thermal and visible spectrum images | |
| CN110443946B (en) | Vending machine, and method and device for identifying types of articles | |
| CN109671227B (en) | Intelligent container consumption behavior identification method and device, storage medium and electronic equipment | |
| JP2018190120A (en) | Article storage device and article management system | |
| CN108557364B (en) | Automatic inventory making method and device | |
| CN111126384A (en) | Commodity classification system and method based on feature fusion | |
| CN106383841A (en) | Purchasing method and apparatus based on picture identification | |
| JP7475000B2 (en) | Storage/Receipt Management Device, Storage/Receipt Management System, and Storage/Receipt Management Method | |
| Achakir et al. | An automated AI-based solution for out-of-stock detection in retail environments | |
| CN115240115A (en) | Visual SLAM loop detection method combining semantic features and bag-of-words model | |
| CN114299388A (en) | Article information specifying method, showcase, and storage medium | |
| CN107032036A (en) | Control the method and automatic vending warehouse in automatic vending warehouse |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |