CN111882606A - Goods shelf commodity layering method based on deep learning - Google Patents
Goods shelf commodity layering method based on deep learning Download PDFInfo
- Publication number
- CN111882606A CN111882606A CN202010626453.0A CN202010626453A CN111882606A CN 111882606 A CN111882606 A CN 111882606A CN 202010626453 A CN202010626453 A CN 202010626453A CN 111882606 A CN111882606 A CN 111882606A
- Authority
- CN
- China
- Prior art keywords
- shelf
- commodity
- training
- model
- shelf layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 17
- 238000013135 deep learning Methods 0.000 title claims abstract description 14
- 238000001514 detection method Methods 0.000 claims abstract description 34
- 230000011218 segmentation Effects 0.000 claims abstract description 25
- 238000005286 illumination Methods 0.000 claims abstract description 10
- 238000002372 labelling Methods 0.000 claims description 6
- 238000003709 image segmentation Methods 0.000 claims description 3
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical group [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 abstract description 7
- 239000010931 gold Substances 0.000 abstract description 7
- 229910052737 gold Inorganic materials 0.000 abstract description 7
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 238000003704 image resize Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000007689 inspection Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Abstract
The invention discloses a goods shelf commodity layering method based on deep learning, and relates to the technical field of artificial intelligence and machine vision deep learning. The method of the invention comprises the following steps: s1, obtaining shelf pictures with different angles, different illumination and different resolutions; s2, training a commodity detection model; s3, training a shelf layer segmentation model; s4, detecting the shelf image to obtain the position frame and the category information of the commodity; s5, carrying out shelf layer segmentation on the shelf image to obtain the region of the shelf layer; s6, acquiring a minimum external rectangle of the shelf layer, and then acquiring a shelf line; and S7, obtaining the center point of the commodity position frame, and then judging the shelf layer where the commodity position frame is located. The invention can automatically layer the commodities on the goods shelf to obtain the number of the goods shelf layers where the commodities are positioned, and further can judge whether the commodities are displayed on the gold position. The method is efficient and accurate, and replaces a manual mode which is low in efficiency and easy to make mistakes.
Description
Technical Field
The invention belongs to the technical field of artificial intelligence and machine vision deep learning, and particularly relates to a goods shelf commodity layering method based on deep learning.
Background
In a new retail age, in order to better control the selling condition of the off-line commodities, retail manufacturers generally send out an industry substitute to go to a supermarket to take a picture for inspection, and then examine the display mode of the commodities. Wherein whether the commodity is displayed in a gold position is very important. "gold locations" generally refer to shelf locations that are easily visible and accessible to consumers, such as 5-tier shelves where gold locations are generally the second and third tiers (from top to bottom).
Judging whether the commodity is displayed in the gold position, firstly, layering the goods shelf, and then, acquiring the layer number of the goods shelf where the commodity is positioned. At present, the information is mainly acquired by checking the pictures by naked eyes, however, for massive pictures and a large number of commodities, the mode is often inefficient and is easy to make mistakes. Therefore, in order to solve the problems, the method for layering the goods on the shelf based on deep learning is significant.
Disclosure of Invention
The invention provides a goods shelf goods layering method based on deep learning, and solves the problems of low efficiency and high possibility of errors of a manual examination mode.
In order to solve the technical problems, the invention is realized by the following technical scheme:
the invention relates to a goods shelf commodity layering method based on deep learning, which comprises the following steps of;
s1, obtaining shelf pictures with different angles, different illumination and different resolutions;
s2, training a commodity detection model, specifically:
s21, collecting shelf commodity pictures with different angles and illumination;
s22, manually marking and training a weaker commodity detection model MO on a part of commodity pictures;
s23, pre-labeling the full data by using the commodity detection model MO, and manually adjusting;
s24, training a commodity detection model M1 by using full data;
s3, training a shelf layer segmentation model, specifically:
s31, collecting shelf commodity pictures with different angles and illumination;
s32, manually marking and training a weak shelf layer segmentation model M2 on part of the commodity pictures;
s33, pre-labeling the full data by using a shelf layer segmentation model M2, and manually adjusting;
s34, training a shelf layer segmentation model M3 by using full data;
s4, detecting the shelf image to obtain the position frame and the category information of the commodity;
s5, carrying out shelf layer segmentation on the shelf image to obtain the region of the shelf layer;
s6, acquiring a minimum external rectangle of the shelf layer, and then acquiring a shelf line;
and S7, obtaining the center point of the commodity position frame, and then judging the shelf layer where the commodity position frame is located.
Further, the commodity detection model MO and the commodity detection model M1 both use a target detection model based on RetinaNet.
Further, the shelf-level segmentation model M2 and the shelf-level segmentation model M3 both use a mask-RCNN image-based segmentation model.
Compared with the prior art, the invention has the following beneficial effects:
the invention can automatically layer the commodities on the goods shelf to obtain the number of the goods shelf layers where the commodities are positioned, and further can judge whether the commodities are displayed on the gold position. The method is efficient and accurate, and replaces a manual mode which is low in efficiency and easy to make mistakes.
Of course, it is not necessary for any product in which the invention is practiced to achieve all of the above-described advantages at the same time.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a shelf commodity layering method based on deep learning according to the present invention;
FIG. 2 is a diagram of the overall steps of the method of the present invention based on FIG. 1;
FIG. 3 is a substep diagram of step S2 in FIG. 2;
fig. 4 is a partial step diagram of step S3 in fig. 2.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1-4, a shelf goods layering method based on deep learning according to the present invention includes the following steps;
s1, obtaining shelf pictures with different angles, different illumination and different resolutions;
s2, training a commodity detection model, specifically:
s21, collecting shelf commodity pictures with different angles and illumination;
s22, manually marking and training a weaker commodity detection model MO on a part of commodity pictures;
s23, pre-labeling the full data by using the commodity detection model MO, and manually adjusting;
s24, training a commodity detection model M1 by using full data;
s3, training a shelf layer segmentation model, specifically:
s31, collecting shelf commodity pictures with different angles and illumination;
s32, manually marking and training a weak shelf layer segmentation model M2 on part of the commodity pictures;
s33, pre-labeling the full data by using a shelf layer segmentation model M2, and manually adjusting;
s34, training a shelf layer segmentation model M3 by using full data;
s4, detecting the shelf image to obtain the position frame and the category information of the commodity;
s5, carrying out shelf layer segmentation on the shelf image to obtain the region of the shelf layer;
s6, acquiring a minimum external rectangle of the shelf layer, and then acquiring a shelf line;
and S7, obtaining the center point of the commodity position frame, and then judging the shelf layer where the commodity position frame is located.
The commodity detection model MO and the commodity detection model M1 both adopt a target detection model based on RetinaNet, and the model can give consideration to both detection precision and detection speed. The RetinaNet uses the Focal local to well balance the foreground target and the background target, so that the detector can be more concentrated on the target which is difficult to distinguish, and the detection precision is improved; meanwhile, due to the single-stage design framework of the single-stage design framework, the high FPS (up to 15FPS on the Nvidia Tesla V100) can be achieved.
The target detection model based on RetinaNet specifically comprises training setting, forward setting and an output result of the model;
RetinaNet training setup:
1. image resize is carried out according to the long edge 1333 pixels and the short edge 800 pixels;
the Batch size is set to 2;
backbone is resnet50, using imageNet based weights;
anchor uses default settings;
5. the learning rate is initialized to 0.01, and the attenuation is 0.1 every 10 epochs;
step 6 is set to 10000 steps, epoch is set to 50
RetinaNet forward setting:
1. image resize is carried out according to the long edge 1333 pixels and the short edge 800 pixels;
2. the detection box score threshold is set to 0.5;
3. the detection box NMS threshold is set to 0.5;
the output result of the RetinaNet model is divided into two parts, namely a detection frame of an object, wherein the detection frame is a list and the elements are [ x1, y1, x2, y2, score ]; the class of objects is a list with elements [ class _1_ pro, class _2_ pro, class _3_ pro,.., class _ n _ pro ] (probability vector for each class). And taking the maximum value of the class probability vector of the object, wherein the corresponding index is the prediction class number.
The shelf layer segmentation model M2 and the shelf layer segmentation model M3 both adopt a mask-RCNN image segmentation model, and the model can segment the shelf layer accurately at the pixel level. Since the shelves are usually inclined at a certain angle, the conventional detection model cannot obtain the precise contour of the shelf layer, so that a complete contour of the shelf layer must be obtained by using a segmentation model, and then an inclined rectangular frame of the shelf layer is obtained, so as to further calculate the shelf line.
The mask-RCNN-based image segmentation model specifically comprises training setting, forward setting and an output result of the model;
mask-RCNN training settings:
1. carrying out image resize according to 1024 pixels on the long side and 800 pixels on the short side;
the Batch size is set to 2;
backbone is resnet50, using imageNet based weights;
anchor uses default settings;
5. the learning rate is initialized to 0.01, and the attenuation is 0.1 every 10 epochs;
step is set as 10000 steps, epoch is set as 50;
mask-RCNN Forward setup:
1. carrying out image resize according to 1024 pixels on the long side and 800 pixels on the short side;
the output result of the mask-RCNN model is divided into three parts: object detection box, object mask, object class. The information of the object detection frame and the category information are the same as those of the RetinaNet. For each piece of mask information of the RoI, the output dimension is Km m. Wherein K denotes encoding m x m images with K binary classes of masks, each mask having K classes;
after the contour of the shelf layer is obtained, the minimum circumscribed rectangle of each shelf layer is fitted, then the middle points are taken in the vertical direction and connected to obtain the shelf line. Then, according to the detection frame information of each commodity, the central point of each commodity is calculated, and then the shelf layer where each commodity is located can be obtained through the distance from the point to the straight line.
Compared with the prior art, the invention has the following beneficial effects:
the invention can automatically layer the commodities on the goods shelf to obtain the number of the goods shelf layers where the commodities are positioned, and further can judge whether the commodities are displayed on the gold position. The method is efficient and accurate, and replaces a manual mode which is low in efficiency and easy to make mistakes.
The preferred embodiments of the invention disclosed above are intended to be illustrative only. The preferred embodiments are not intended to be exhaustive or to limit the invention to the precise embodiments disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best utilize the invention. The invention is limited only by the claims and their full scope and equivalents.
Claims (3)
1. A goods shelf layering method based on deep learning is characterized by comprising the following steps;
s1, obtaining shelf pictures with different angles, different illumination and different resolutions;
s2, training a commodity detection model, specifically:
s21, collecting shelf commodity pictures with different angles and illumination;
s22, manually marking and training a weaker commodity detection model MO on a part of commodity pictures;
s23, pre-labeling the full data by using the commodity detection model MO, and manually adjusting;
s24, training a commodity detection model M1 by using full data;
s3, training a shelf layer segmentation model, specifically:
s31, collecting shelf commodity pictures with different angles and illumination;
s32, manually marking and training a weak shelf layer segmentation model M2 on part of the commodity pictures;
s33, pre-labeling the full data by using a shelf layer segmentation model M2, and manually adjusting;
s34, training a shelf layer segmentation model M3 by using full data;
s4, detecting the shelf image to obtain the position frame and the category information of the commodity;
s5, carrying out shelf layer segmentation on the shelf image to obtain the region of the shelf layer;
s6, acquiring a minimum external rectangle of the shelf layer, and then acquiring a shelf line;
and S7, obtaining the center point of the commodity position frame, and then judging the shelf layer where the commodity position frame is located.
2. The deep learning-based goods shelf layering method according to claim 1, wherein the goods detection model MO and the goods detection model M1 both use a RetinaNet-based target detection model.
3. The deep learning-based shelf commodity layering method according to claim 1, wherein the shelf layer segmentation model M2 and the shelf layer segmentation model M3 both use a mask-RCNN image segmentation model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010626453.0A CN111882606A (en) | 2020-07-01 | 2020-07-01 | Goods shelf commodity layering method based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010626453.0A CN111882606A (en) | 2020-07-01 | 2020-07-01 | Goods shelf commodity layering method based on deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111882606A true CN111882606A (en) | 2020-11-03 |
Family
ID=73150001
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010626453.0A Pending CN111882606A (en) | 2020-07-01 | 2020-07-01 | Goods shelf commodity layering method based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111882606A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112446437A (en) * | 2020-12-11 | 2021-03-05 | 上海品览数据科技有限公司 | Goods shelf commodity specification identification method based on machine vision |
| CN112990095A (en) * | 2021-04-13 | 2021-06-18 | 广州市玄武无线科技股份有限公司 | Commodity display analysis method, commodity display analysis device, commodity display analysis equipment and storage medium |
| CN114972505A (en) * | 2022-04-29 | 2022-08-30 | 弥费实业(上海)有限公司 | Position recognition system |
| CN115359065A (en) * | 2022-10-24 | 2022-11-18 | 广州市玄武无线科技股份有限公司 | Inclination detection method and device for plane arrangement image |
| CN115909358A (en) * | 2022-07-27 | 2023-04-04 | 广州市玄武无线科技股份有限公司 | Commodity specification identification method and device, terminal equipment and computer storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108573279A (en) * | 2018-03-19 | 2018-09-25 | 精锐视觉智能科技（深圳）有限公司 | Image labeling method and terminal device |
| CN108985293A (en) * | 2018-06-22 | 2018-12-11 | 深源恒际科技有限公司 | A kind of image automation mask method and system based on deep learning |
| CN109543527A (en) * | 2018-10-19 | 2019-03-29 | 北京陌上花科技有限公司 | For the commodity detection method of unmanned shelf, device and retail terminal |
| CN109685154A (en) * | 2018-12-29 | 2019-04-26 | 天津链数科技有限公司 | A kind of method of image data mark label |
| CN110059753A (en) * | 2019-04-19 | 2019-07-26 | 北京朗镜科技有限责任公司 | Model training method, interlayer are every recognition methods, device, equipment and medium |
| CN110245580A (en) * | 2019-05-24 | 2019-09-17 | 北京百度网讯科技有限公司 | A kind of method, apparatus of detection image, equipment and computer storage medium |
| CN110472486A (en) * | 2019-07-03 | 2019-11-19 | 北京三快在线科技有限公司 | A kind of shelf obstacle recognition method, device, equipment and readable storage medium storing program for executing |
-
2020
- 2020-07-01 CN CN202010626453.0A patent/CN111882606A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108573279A (en) * | 2018-03-19 | 2018-09-25 | 精锐视觉智能科技（深圳）有限公司 | Image labeling method and terminal device |
| CN108985293A (en) * | 2018-06-22 | 2018-12-11 | 深源恒际科技有限公司 | A kind of image automation mask method and system based on deep learning |
| CN109543527A (en) * | 2018-10-19 | 2019-03-29 | 北京陌上花科技有限公司 | For the commodity detection method of unmanned shelf, device and retail terminal |
| CN109685154A (en) * | 2018-12-29 | 2019-04-26 | 天津链数科技有限公司 | A kind of method of image data mark label |
| CN110059753A (en) * | 2019-04-19 | 2019-07-26 | 北京朗镜科技有限责任公司 | Model training method, interlayer are every recognition methods, device, equipment and medium |
| CN110245580A (en) * | 2019-05-24 | 2019-09-17 | 北京百度网讯科技有限公司 | A kind of method, apparatus of detection image, equipment and computer storage medium |
| CN110472486A (en) * | 2019-07-03 | 2019-11-19 | 北京三快在线科技有限公司 | A kind of shelf obstacle recognition method, device, equipment and readable storage medium storing program for executing |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112446437A (en) * | 2020-12-11 | 2021-03-05 | 上海品览数据科技有限公司 | Goods shelf commodity specification identification method based on machine vision |
| CN112990095A (en) * | 2021-04-13 | 2021-06-18 | 广州市玄武无线科技股份有限公司 | Commodity display analysis method, commodity display analysis device, commodity display analysis equipment and storage medium |
| CN112990095B (en) * | 2021-04-13 | 2021-09-14 | 广州市玄武无线科技股份有限公司 | Commodity display analysis method, commodity display analysis device, commodity display analysis equipment and storage medium |
| CN114972505A (en) * | 2022-04-29 | 2022-08-30 | 弥费实业(上海)有限公司 | Position recognition system |
| CN115909358A (en) * | 2022-07-27 | 2023-04-04 | 广州市玄武无线科技股份有限公司 | Commodity specification identification method and device, terminal equipment and computer storage medium |
| CN115909358B (en) * | 2022-07-27 | 2024-02-13 | 广州市玄武无线科技股份有限公司 | Commodity specification identification method, commodity specification identification device, terminal equipment and computer storage medium |
| CN115359065A (en) * | 2022-10-24 | 2022-11-18 | 广州市玄武无线科技股份有限公司 | Inclination detection method and device for plane arrangement image |
| CN115359065B (en) * | 2022-10-24 | 2023-04-07 | 广州市玄武无线科技股份有限公司 | Inclination detection method and device for plane arrangement image |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111882606A (en) | Goods shelf commodity layering method based on deep learning | |
| CN110163640A (en) | A kind of method and computer equipment of product placement in video | |
| CN111898406B (en) | Face detection method based on focus loss and multitask cascade | |
| CN113362329B (en) | Method for training focus detection model and method for recognizing focus in image | |
| US8422783B2 (en) | Methods and systems for region-based up-scaling | |
| CN102473282B (en) | External light glare assessment device, line of sight detection device and external light glare assessment method | |
| CN112446437A (en) | Goods shelf commodity specification identification method based on machine vision | |
| CN111931836A (en) | Method and device for acquiring neural network training image | |
| CN111681273A (en) | Image segmentation method and device, electronic equipment and readable storage medium | |
| CN111507976A (en) | Defect detection method and system based on multi-angle imaging | |
| CN114155527A (en) | Scene text recognition method and device | |
| CN113869211A (en) | Automatic image annotation and automatic annotation quality evaluation method and system | |
| CN111027554A (en) | System and method for accurately detecting and positioning commodity price tag characters | |
| CN115830004A (en) | Surface defect detection method, device, computer equipment and storage medium | |
| CN112989995A (en) | Text detection method and device and electronic equipment | |
| CN112329522A (en) | Goods shelf goods fuzzy detection method based on deep learning and image processing | |
| CN114494823A (en) | Commodity identification, detection and counting method and system in retail scene | |
| CN110738123B (en) | Method and device for identifying densely displayed commodities | |
| CN115908988B (en) | Defect detection model generation method, device, equipment and storage medium | |
| CN109815896B (en) | Parameter-configurable virtual goods channel dividing method for intelligent sales counter | |
| CN117037132A (en) | Ship water gauge reading detection and identification method based on machine vision | |
| CN113763384B (en) | Defect detection method and defect detection device in industrial quality inspection | |
| CN115205155A (en) | Distorted image correction method and device and terminal equipment | |
| CN115546141A (en) | Small sample Mini LED defect detection method and system based on multi-dimensional measurement | |
| CN111489332B (en) | Multi-scale IOF random cutting data enhancement method for target detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201103 |
|
| RJ01 | Rejection of invention patent application after publication |