CN111898417A - Container system, goods detection device and method - Google Patents

Container system, goods detection device and method Download PDF

Info

Publication number
CN111898417A
CN111898417A CN202010553634.5A CN202010553634A CN111898417A CN 111898417 A CN111898417 A CN 111898417A CN 202010553634 A CN202010553634 A CN 202010553634A CN 111898417 A CN111898417 A CN 111898417A
Authority
CN
China
Prior art keywords
goods
abnormal
target
detection
label
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010553634.5A
Other languages
Chinese (zh)
Other versions
CN111898417B (en
Inventor
钟华堡
张帆
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xiamen Hualian Electronics Co Ltd
Original Assignee
Xiamen Hualian Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xiamen Hualian Electronics Co Ltd filed Critical Xiamen Hualian Electronics Co Ltd
Priority to CN202010553634.5A priority Critical patent/CN111898417B/en
Publication of CN111898417A publication Critical patent/CN111898417A/en
Application granted granted Critical
Publication of CN111898417B publication Critical patent/CN111898417B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computing arrangements based on specific mathematical models
    • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Probability & Statistics with Applications (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Mathematical Optimization (AREA)
  • Mathematical Physics (AREA)
  • Mathematical Analysis (AREA)
  • Algebra (AREA)
  • Pure & Applied Mathematics (AREA)
  • Computing Systems (AREA)
  • Multimedia (AREA)
  • Computational Mathematics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a container system, a goods detection device and a method, wherein the method comprises the following steps: carrying out multi-target detection on the goods image by adopting a deep learning multi-target detection algorithm through a goods detection device to obtain the deep visual characteristics and the prediction information of the target goods; obtaining an abnormal prediction probability set of the target goods according to the depth visual features by using a classification method; respectively judging abnormal prediction probability set PiWhether the abnormality prediction probability of the K kinds of abnormalities is greater than a threshold value; and adding the abnormal information of the kth abnormality with the abnormal prediction probability larger than the threshold value into the label information to generate abnormal prediction information. According to the invention, the abnormal goods are identified based on the pre-trained detection model, and the abnormal goods are processed in time through the server and the interactive terminal.

Description

Container system, goods detection device and method
Technical Field
The embodiment of the invention relates to the technical field of image recognition, in particular to a container system, a goods detection device and a goods detection method.
Background
The existing visual unmanned sales counter adopts a dynamic image recognition technology, and counts and generates orders by recognizing the dynamic in-out quantity and type of goods at the inlet of the sales counter; or a static image recognition technology is adopted to generate the order by recognizing and counting the quantity difference of goods in the container before and after the opening and closing of the container door.
The existing image recognition technology usually adopts a multi-target detection technology (such as fast-RCNN, SSD) based on deep learning, and trains a target detection module by using a large amount of pre-given labeled image data, wherein the target detection module can predict a position frame and a category of each target item in an item image of a sales counter, so as to be used for generating a shopping order of a user.
In practical application, when goods are maliciously consumed by customers or the goods are unpacked and originally packaged, or the goods are placed in a mess to cause the key characteristics of the goods to be shielded or the goods to fall down, the image recognition technology cannot accurately recognize the goods at the moment. As shown in fig. 1 to 3, if a damaged article is mixed into a normal article, the damaged article cannot be effectively distinguished by using the existing method, and the damaged article is still considered as an inventory article; as shown in fig. 4, whether the existing method can identify the severely occluded goods and whether the existing method needs to identify the goods is a controversial problem; as shown in fig. 5 and 6, the existing method can not prompt the manager about the falling of the goods, and can be changed into a serious shielding situation at any time. In addition, multiple such anomaly situations may occur simultaneously, at which time it is not feasible to classify anomalies using the item detection techniques described above.
Aiming at the problems, the existing goods detection technology can not effectively detect abnormal goods, so that the management of an unmanned sales counter is difficult, the risk of goods loss can not be ensured, and even serious food safety problems can be caused.
Disclosure of Invention
The technical problem mainly solved by the embodiment of the invention is to provide a container system and a goods detection method thereof, which can accurately identify and count the goods abnormal conditions in a sales container so as to ensure the effective management of the goods of an unmanned sales container.
In order to solve the technical problems, the invention adopts a technical scheme that: the goods cabinet system comprises a goods cabinet, a goods detection device, a server, an interactive terminal and a detection model training device, wherein the goods detection device is used for collecting goods images of goods in the goods cabinet, and the detection model training device is used for training and detectingMeasuring the model and generating a detection model with preset model parameters; the goods detection device is also used for carrying out goods detection on the goods image by using the detection model with the preset model parameters to generate a detection result; wherein the article detection device includes a first processing unit including: the goods detection module is used for carrying out multi-target detection on the goods image by adopting a multi-target detection algorithm based on deep learning to obtain the depth visual characteristics and the prediction information of the target goods; the prediction information comprises first image data and first label information of a target goods, and the first label information comprises category prediction and position frame prediction of the target goods; an anomaly classification module to: obtaining an abnormal prediction probability set P of the target goods according to the depth visual features by using a classification methodi＝{pi,k|k∈N+,K∈N+And K is more than or equal to 1 and less than or equal to K }; wherein K is the number of the abnormal category, K is the total number of the abnormal category, i is the number of the target goods, i belongs to N+，N+Is a positive integer; respectively judging the abnormal prediction probability set PiThe anomaly prediction probability p of K anomalies in (1)i,kWhether greater than a threshold; and predicting the probability of abnormality pi,kAnd adding the abnormal information of the kth abnormity which is larger than the threshold value into the label information to generate abnormity prediction information.
The detection model training device comprises a storage unit for storing training images and a second processing unit, wherein the second processing unit comprises: the marking module is used for carrying out multi-target detection marking on the training image to obtain a first marking data set formed by marking values of target goods; the labeling value comprises second image data and second label information of the target goods, and the second label information comprises category labeling and position frame labeling of the target goods; the abnormality labeling module is used for adding the abnormality label of the kth abnormality in the label value of each abnormal target goods in the first label data set to form an abnormality label value, so as to obtain a second label data set containing the abnormality label value; a parameter training module comprising: the loss function calculation submodule is used for calculating an abnormal classification loss function weighting coefficient; the system is also used for calculating a joint loss function of a detection model according to the forecast information of the target goods generated by the goods detection module, the abnormal forecast information generated by the abnormal classification module, the second labeled data set and the abnormal classification loss function weighting coefficient; the model parameter setting submodule is used for calculating a partial derivative of the current model parameter of the detection model based on a chain rule and the combined loss function when the loss function calculating submodule judges that the combined loss function is not converged, and subtracting the partial derivative from the current model parameter to obtain a new current model parameter of the detection model; and the first processing unit is used for finishing the training when the loss function calculation submodule judges that the combined loss function is converged, and using the new current model parameters for actual goods detection.
Wherein the loss function calculation submodule is configured to: calculating an abnormal classification loss function weighting coefficient by using the formula (2); wherein λ is a decimal of 0, x is an image in the data set, and a is a set formed by images with abnormal labels in the second labeled data set;
Figure BDA0002543431660000031
calculating a goods detection loss function loss of the goods detection modulegeneral(ii) a Calculating an abnormality classification loss function loss of the abnormality classification module using the formulas (4) to (8)anomaly(ii) a Wherein l (i) is used to calculate the loss of a single predicted target; iouiBbox representing ith target item in the training imageiThe maximum overlapping rate of the position frames of all the target goods with the abnormal labels is obtained; y isi,kAn abnormity label for indicating whether the ith target item has the k type abnormity, if yi,kIf the value is 1, the abnormal mark exists, and if yi,kIf the value is 0, the abnormal label is not existed; wherein i ∈ N+，I∈N+I is more than or equal to 1 and less than or equal to I, I is the number of the target goods, I is the total number of the target goods, N+To be just neatCounting; gtboxjA position frame "bbox" showing the jth target item with abnormal annotationiA location box representing a predicted ith target item; gtclsj,kIndicating whether the k-th abnormal label of the jth abnormal label exists or not, idiIndicating the number of the target goods with the abnormal label with the maximum overlapping rate with the ith target goods i, wherein j is the number of the target goods with the abnormal label, and j belongs to N+；
Figure BDA0002543431660000041
Figure BDA0002543431660000042
Figure BDA0002543431660000043
Figure BDA0002543431660000044
Figure BDA0002543431660000046
According to the goods detection loss function lossgeneralThe anomaly classification loss function lossanomalyAnd the weighting coefficient lambda of the abnormal classification loss function, and the joint loss function loss is calculated by using the formula (9)
loss＝lossgeneral+λ·lossanomaly(9)。
Wherein the loss function calculation sub-module is further configured to utilize the value calculated by the equation (3):
Figure BDA0002543431660000045
wherein n is the total number of all target items in the second annotation data set, m1For the abnormal target goods in the set ANumber of (2), m2The number of normal target items in the collection a.
Wherein the anomaly classification module is used for calculating an anomaly prediction probability set P of the target goods by utilizing the formula (1)i(ii) a Where x is the input image, the function F represents the candidate object extractor, hiA feature representing an ith prediction target; function G represents the item Classification and predictive Box component, { clsi,bboxiExpressing the goods category and the position frame predicted by the ith prediction target; function H represents an anomaly classification module, PiSet of probabilities representing that ith item is predicted to be in respective exception categories
Figure BDA0002543431660000051
In order to solve the technical problem, the invention adopts another technical scheme as follows: there is provided an article detection method applied to the container system as described above, the method including: the detection model training device generates a detection model with set model parameters; the goods detection device utilizes the detection model with the set model parameters to carry out goods detection on the collected goods images of the goods in the container so as to generate a detection result; the method specifically comprises the following steps: the goods detection device adopts a multi-target detection algorithm based on deep learning to carry out multi-target detection on the goods image to obtain the depth visual characteristics and the prediction information of a target goods i; the prediction information comprises first image data and first label information of a target goods, and the first label information comprises category prediction and position frame prediction of the target goods; the goods detection device obtains an abnormal prediction probability set P of the target goods according to the depth visual features by using a classification methodi＝{pi,k|k∈N+,K∈N+And K is more than or equal to 1 and less than or equal to K }; wherein K is the number of the abnormal category, K is the total number of the abnormal category, i is the number of the target goods, i belongs to N+，N+Is a positive integer; the goods detection device respectively judges the abnormal prediction probability sets PiThe anomaly prediction probability p of K anomalies in (1)i,kWhether greater than a threshold; the above-mentionedThe goods detection device predicts the abnormality probability pi,kAnd adding the abnormal information of the kth abnormity which is larger than the threshold value into the label information to generate abnormity prediction information.
The "detection model training apparatus generates a detection model having set model parameters", and specifically includes: the detection model training device carries out multi-target detection and labeling on the training image to obtain a first labeling data set formed by labeling values of target goods; the labeling value comprises second image data and second label information of the target goods, and the second label information comprises category labeling and position frame labeling of the target goods; the detection model training device adds the abnormal labels of the kth abnormality in the label value of each abnormal target goods j in the first label data set to form abnormal label values, and a second label data set comprising the abnormal labels is obtained; the detection model training device sends the training image to the goods detection device, and prediction information and abnormal prediction information of the target goods i are generated through a detection model with set model parameters; the detection model training device calculates an abnormal classification loss function weighting coefficient; the detection model training device calculates a joint loss function of the detection model according to the prediction information and the abnormal prediction information of the target goods generated by the goods detection device, the second labeled data set and the abnormal classification loss function weighting coefficient; when the detection model training device judges that the joint loss function is not converged, calculating a partial derivative of the current model parameter of the detection model based on a chain rule and the joint loss function, and subtracting the partial derivative from the current model parameter to obtain a new current model parameter of the detection model; and when the joint loss function is judged to be converged, finishing the training.
The "calculating an abnormal classification loss function weighting coefficient by the detection model training device" specifically includes: the detection model training device calculates an abnormal classification loss function weighting coefficient by using a formula (2); wherein λ is a decimal of 0, x is a lower image in the data set, and a is a set formed by images with abnormal labels in the second labeled data set;
Figure BDA0002543431660000061
the method for calculating the combined loss function of the detection model by the detection model training device according to the prediction information and the abnormal prediction information of the target goods generated by the goods detection device, the second labeled data set and the abnormal classification loss function weighting coefficient specifically comprises the following steps: calculating the goods detection loss function lossgeneral(ii) a Calculating an abnormality classification loss function loss of the abnormality classification module using the formulas (4) to (8)anomaly(ii) a Wherein l (i) is used to calculate the loss of a single predicted target; iouiBbox representing ith target item in the training imageiThe maximum overlapping rate of the position frames of all the target goods with the abnormal labels is obtained; y isi,kAn abnormal label showing whether the ith target item i has the k-th abnormality or not is given, if yi,kIf the value is 1, the abnormal mark exists, and if yi,kIf the value is 0, the abnormal label is not existed; gtboxjA position frame, bbox, indicating the jth target item with abnormal labeliA location box representing a predicted ith target item; gtclsj,kIndicating whether the k-th abnormal label of the jth abnormal label exists or not, idiIndicating the number of the target goods with the abnormal label with the maximum overlapping rate with the ith target goods i, wherein j is the number of the target goods with the abnormal label, and j belongs to N+；
Figure BDA0002543431660000062
Figure BDA0002543431660000071
Figure BDA0002543431660000072
Figure BDA0002543431660000073
Figure BDA0002543431660000075
According to the goods detection loss function lossgeneralThe anomaly classification loss function lossanomalyAnd the weighting coefficient lambda of the abnormal classification loss function, and the joint loss function loss is calculated by using the formula (9)
loss＝lossgeneral+λ·lossanomaly(9)。
Before the detecting model training device calculates the weighting coefficient of the abnormal classification loss function by using the formula (2), the method further includes: the detection model training means uses the value calculated by the formula (3);
Figure BDA0002543431660000074
wherein n is the total number of all target items in the second annotation data set, m1Number of anomalous target items in set A, m2The number of normal target items in the collection a.
Wherein the goods detection device obtains the abnormal prediction probability set P of the target goods according to the depth visual characteristics by using a classification methodi＝{pi,k|k∈N+,K∈N+And K is more than or equal to 1 and less than or equal to K, the method specifically comprises the following steps: the goods detection device obtains an abnormal prediction probability set P of the target goods according to the depth visual characteristics by using a formula (1)i＝{pi,k|k∈N+,K∈N+And K is more than or equal to 1 and less than or equal to K }; where x is the input image, the function F represents the candidate object extractor, hiA feature representing an ith prediction target; function G represents the item Classification and predictive Box component, { clsi,bboxiIndicating the goods category and the position frame of which the ith prediction target is predicted; function H represents an anomaly classification module, PiSet of probabilities representing that ith item is predicted to be in respective exception categories
Figure BDA0002543431660000081
In order to solve the technical problem, the invention adopts another technical scheme that: there is provided an article detection device including a first processing unit including: the goods detection module is used for carrying out multi-target detection on the goods image by adopting a multi-target detection algorithm based on deep learning to obtain the depth visual characteristics and the prediction information of the target goods; the prediction information comprises first image data of a target item i and first label information, and the first label information comprises category prediction and position frame prediction of the target item; an anomaly classification module to: obtaining an abnormal prediction probability set P of the target goods according to the depth visual features by using a classification methodi＝{pi,k|k∈N+,K∈N+And K is more than or equal to 1 and less than or equal to K }; wherein K is the number of the abnormal category, K is the total number of the abnormal category, i is the number of the target goods, i belongs to N+，N+Is a positive integer; respectively judging the abnormal prediction probability set PiThe anomaly prediction probability p of K anomalies in (1)i,kWhether greater than a threshold; and predicting the probability of abnormality pi,kAnd adding the abnormal information of the kth abnormity which is larger than the threshold value into the label information to generate abnormity prediction information.
In order to solve the technical problems, the invention adopts a technical scheme that: there is provided an article detection method applied to the article detection device as described above, the method including: the goods detection device adopts a multi-target detection algorithm based on deep learning to carry out multi-target detection on the goods image to obtain the depth visual characteristics and the prediction information of the target goods; the prediction information comprises first image data and first label information of a target item i, and the first label information comprises category prediction and position frame prediction of the target item i; the goods detection device obtains the abnormal prediction probability of the target goods according to the depth visual features by using a classification methodSet Pi＝{pi,k|k∈N+,K∈N+And K is more than or equal to 1 and less than or equal to K }; wherein K is the number of the abnormal category, K is the total number of the abnormal category, i is the number of the target goods, i belongs to N+，N+Is a positive integer; the goods detection device respectively judges the abnormal prediction probability set PiThe anomaly prediction probability p of K anomalies in (1)i,kWhether greater than a threshold; the goods detecting apparatus predicts the abnormality probability pi,kAnd adding the abnormal information of the kth abnormity which is larger than the threshold value into the label information to generate abnormity prediction information.
The beneficial effects of the embodiment of the invention are as follows: different from the situation of the prior art, the container system and the goods detection method thereof in the embodiment of the invention can identify abnormal goods based on the pre-trained detection model and process the abnormal goods in time through the interactive terminal and the server.
Drawings
FIG. 1 is a schematic view of a first image of an article in a broken condition;
fig. 2 is a schematic view of a second image in the event of breakage of the article;
FIG. 3 is a third image schematic in the event of a failure of the article;
FIG. 4 is a schematic image of a case where the item is heavily occluded;
FIG. 5 is a schematic image of a situation where the item is heavily occluded and the item is lodging;
FIG. 6 is a schematic image of an article in a collapsed condition;
FIG. 7 is a schematic structural view of a sales counter system according to an embodiment of the present invention;
fig. 8 is a functional block diagram of an article identification device according to a first embodiment of the present invention;
FIG. 9 is a functional block diagram of a test model training apparatus according to an embodiment of the present invention;
FIG. 10 is a schematic flow chart of a method of detecting an item in accordance with one embodiment of the present invention;
FIG. 11 is a flowchart illustrating a method implemented in step S2 shown in FIG. 10;
FIG. 12 is a flowchart illustrating a method implemented in step S22 shown in FIG. 11;
FIG. 13 is a flowchart illustrating a method implemented in step S1 shown in FIG. 10;
fig. 14 is a flowchart illustrating a method implemented in step S14 shown in fig. 13.
Detailed Description
In order to facilitate an understanding of the invention, the invention is described in more detail below with reference to the accompanying drawings and detailed description.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Please refer to fig. 7, which is a schematic structural diagram of a container system 100 according to an embodiment of the present invention. The container system 100 includes a sales container 10, a goods detection device 20, a server 30 storing user data and goods data, an interactive terminal 40, and a detection model training device 50. The goods detection device 20 is in communication connection with the server 30 and the interaction terminal 40, respectively.
The detection model training device 50 is used for training a detection model and generating the detection model with preset model parameters.
The goods detection device 20 is used for collecting goods images of goods in the sales counter, detecting the goods of the goods image of the counter 10 by using the detection model with preset model parameters to generate a detection result, detecting the goods of the input goods training image based on the detection model parameters to generate a detection result, and assisting in finishing the training of the detection model.
The server 30 is configured to query and update the user data and the goods data according to the detection result. Further, the server 30 is further configured to generate prompt information such as goods taking information, payment information, and abnormal information according to the detection result, and send the prompt information to the interactive terminal 40. Specifically, in one embodiment, when a user frequently causes an exception prompt, the server 40 reduces the credit score of the user according to the exception prompt information.
The interactive terminal 40 is configured to interact with the user according to the prompt message generated by the server 30. Further, the interactive terminal 40 is further configured to implement basic interaction such as code scanning or user information collection. Specifically, in an embodiment, the interactive terminal 40 controls a voice broadcaster to play prompt information; in another embodiment, the interactive terminal 40 controls a display to play a prompt message; in another embodiment, the interactive terminal 40 performs code scanning or face scanning recognition to open the cargo cabinet door.
Fig. 8 is a schematic structural diagram of an article detection device 20 according to an embodiment of the present invention. The goods detection device 20 comprises a first processing unit 21 and an image acquisition unit 22 connected with the first processing unit 21, wherein the image acquisition unit 22 is used for acquiring goods images of goods in the container 10. The first processing unit 21 includes an item detection module 211 and an abnormality classification module 212.
The goods detection module 211 is configured to perform multi-target detection on the goods image by using a multi-target detection algorithm (such as fast-RCNN, SSD) based on deep learning to obtain a deep visual feature and prediction information of a target goods i, where i belongs to N+，I∈N+I is more than or equal to 1 and less than or equal to I, I is the number of the target goods, I is the total number of the target goods, N+Is a positive integer. The prediction information comprises first image data of a target item i and first label information, and the first label information comprises class prediction cls of the target item iiAnd location box prediction bboxi. The first image data includes an image file name, and may further include information such as an image size and an image format.
In particular, the item detection module 211 includes a candidate target extractor 2111, an item classification and forecast box component 2112. The candidate target extractor 2111 is operable to map the item image to depth visual features of location areas where a target item may be present. The goods classification and forecast boxThe component 2112 is for predicting cls for the category of the target item i based on the depth vision featureiAnd location box prediction bboxi。
For example, when the fast-RCNN algorithm is employed, the candidate object extractor 221 is configured to extract features in the item image and output a plurality of region proposals (regionproposals) after being region-pooled, each region proposal including the depth visual feature. For example, when the SSD algorithm is used, the candidate target extractor 221 is configured to obtain a multi-scale feature map according to a good image, where features included in each feature pixel point in the multi-scale feature map all represent multiple candidate targets, the number of candidate targets is the number of scale boxes (anchor boxes) of the multi-scale feature map, and each scale box includes the depth visual feature.
The anomaly classification module 212 is configured to:
obtaining an abnormal prediction probability set P of the target goods i according to the depth visual features by using a classification methodi＝{pi,k|k∈N+,K∈N+And K is more than or equal to 1 and less than or equal to K }; wherein K is the number of the abnormal types, and K is the total number of the abnormal types; the abnormality comprises at least one of breakage, severe shielding and lodging;
judging the abnormality prediction probability p of the kth abnormalityi,kWhether it is greater than a threshold t; and
in determining the abnormality prediction probability p of the k-th abnormalityi,kAnd when the number of the k-th abnormity of the target goods i is larger than the threshold value t, determining that the k-th abnormity exists in the target goods i, and adding the abnormity information of the k-th abnormity into the label information to generate abnormity prediction information. In particular, the classification method is a multi-label classification method.
Specifically, the anomaly classification module 212 calculates the set P of anomaly prediction probabilities of the target item i by using the formula (1)i(ii) a Where x is the input image, the function F represents the candidate object extractor, hiA feature representing an ith prediction target; function G represents the item Classification and predictive Box component, { clsi，bboxiExpressing the goods category and the position frame predicted by the ith prediction target; function H represents an anomaly classification module, PiIs shown asThe probability set of i goods predicted to each abnormal category;
Figure BDA0002543431660000111
for example, let K be 3, the abnormality types be 1. breakage, 2. severe occlusion, 3. lodging, and the set P of abnormality prediction probabilities of the target item 11If the threshold t is 0.5, then the anomaly classification module 212 determines that there are 2 nd and 3 rd anomalies for the target item 1, i.e., there is a severe occlusion and lodging. The corresponding abnormality detection information of the target goods 1 is {1.jpg, and the position frame predicts bbox1Class prediction cls1Severe occlusion, lodging }; as another example, assume the anomaly prediction probability set P for the target item 22If the threshold t is 0.5, then the target item 2 does not have any abnormality, the detection information corresponding to the target item 2 is {1.jpg, and the location box predicts bbox2Class prediction cls2}。
Fig. 9 is a schematic structural diagram of a training apparatus for testing models according to an embodiment of the present invention. The detection model training apparatus 50 includes a second processing unit 51 and a storage unit 52, and the storage unit 52 is used for storing training images. The second processing unit 51 comprises an annotation module 511, an anomaly annotation module 512 and a parameter training module 513.
The labeling module 511 is configured to perform multi-target detection labeling on the training image to obtain a first labeling data set formed by labeling values of the target goods i; and the label value comprises second image data and second label information of the target item i, and the second label information comprises category labels and position frame labels of the target item i. The labeling format can be a Pascal VOC format, and the corresponding labeling tool can be labelImg; the annotation format may also be a YOLO format and the corresponding annotation tool may be YOLO _ mark. Other conventional formats and corresponding labeling tools are also possible. Through the labeling tools, a box is dragged for each target item by using a mouse to determine a position box and select a category, and then a label file in a corresponding format is saved and generated. For example, the first labeled data set for each target item is { image filename, location box label, category label }.
The anomaly labeling module 512 is configured to add an anomaly label of a kth anomaly to the label value of each anomalous target item j in the first labeled data set to form an anomaly label value, so as to obtain a second labeled data set including the anomaly label value; wherein j ∈ N+，J∈N+J is more than or equal to 1 and less than or equal to J, J is the number of the target goods with the abnormal labels, J is the total number of the target goods with the abnormal labels, and gtboxjAnd marking the position frame of the jth target item with the abnormal mark.
For example, a training image 0001.jpg is subjected to multi-target detection annotation, the image comprises 3 target goods M1, M2 and M3, wherein abnormal annotations exist in the target goods M1 and M2, and no abnormal annotation exists in M3, and then the annotation data set of each target goods is as follows:
m1: {0001.jpg, position frame 1, item category 1, damaged, heavily occluded, lodging };
m2: {0001.jpg, position box 2, item category 2, severe occlusion, lodging };
m3: {0001.jpg, location box 3, item category 3 }.
The parameter training module 513 includes:
a loss function calculation sub-module 5131 for calculating an anomaly classification loss function weighting coefficient λ; and is further configured to calculate a joint loss function of the detection model according to the forecast information of the target item i generated by the item detection module 211, the anomaly forecast information generated by the anomaly classification module 212, the second labeled data set generated by the anomaly labeling module 512, and the weighting coefficient λ of the anomaly classification loss function, and determine whether the joint loss function is converged.
A model parameter setting submodule 5132, configured to, when the loss function calculating submodule 5132 determines that the combined loss function is not converged, calculate a partial derivative of the current model parameter of the detection model based on a chain rule and the combined loss function, and subtract the partial derivative from the current model parameter to obtain a new current model parameter of the detection model; and is further used for ending the training when the loss function calculation submodule 5132 judges that the joint loss function converges.
Further, the loss function calculation sub-module 5131 is configured to calculate a goods detection loss function of the goods detection module 211 and an abnormal classification loss function of the abnormal classification module 212, and calculate the joint loss function according to the goods detection loss function and the abnormal classification loss function.
Specifically, the loss function calculation sub-module 5132 calculates the anomaly classification loss function weighting coefficient λ using the following formula (2):
Figure BDA0002543431660000131
among them, a decimal number close to 0 is used. x is an image in the data set, and A is a set of images with abnormal annotations in the second annotation data set.
In practical situations, the number of abnormal target goods is much smaller than that of normal target goods, so that the problem of unbalanced category can be faced in the application scene of the invention, and the common solutions are as follows: oversampling, undersampling, cost sensitive learning, Focal local, and the like. The invention adopts an undersampling method to undersample the quantity of normal target goods with more quantity, divides a training set into A, B sets and calculates the value of lambda. Under-sampling methods are commonly used for classification tasks, but the scene of the invention is a detection task, namely, each image contains a plurality of targets and a plurality of categories, and the direct use of the under-sampling method has problems: in general, data acquisition is of a normal target, that is, goods in one image are normal and abnormal, but the goods in the set A may be abnormal, in this case, if the goods in one image is equal to 0, the abnormal classification module can never learn to distinguish the normal target, so the invention also adds a calculation formula of a 'cost-sensitive learning method' to solve the problem.
Specifically, the loss function calculation sub-module 5132 further uses a value calculated by the following equation (3):
Figure BDA0002543431660000141
wherein n is the total number of all target items in the second annotation data set, m1Number of anomalous target items in set A, m2The number of normal target items in the collection a.
The loss function calculation sub-module 5132 calculates the goods detection loss function loss of the goods detection modulegeneral，lossgeneralIs defined by a specific multi-target detection algorithm (as defined by fast-RCNN, SSD);
the loss function calculation sub-module 5132 calculates the abnormality classification loss function loss of the abnormality classification module 212 using the following equations (4) to (8)anomaly：
Figure BDA0002543431660000142
Figure BDA0002543431660000143
Figure BDA0002543431660000144
Figure BDA0002543431660000145
Figure BDA0002543431660000146
Wherein L (i) represents a single item target hiLoss function of, iouiBbox representing the ith target item in the training imageiMaximum overlapping rate, y, with all target goods position frames with abnormal labelsi，kAn abnormal label showing whether the ith target item i has the k-th abnormality or not is given, if ykIf the value is 1, the abnormal label exists; if ykIf 0, it means that the abnormal label is not storedAt this point. gtboxiA position frame, bbox, indicating the jth target item with abnormal labeliA location box representing a predicted ith target item; gtclsj,kIndicating whether the k-th abnormal label of the jth abnormal label exists or not, idiAnd indicating the number of the target goods with the abnormal labels, which has the maximum overlapping rate with the ith target goods i.
Further, the loss function calculation sub-module 5132 calculates the joint loss function loss using the following equation (9):
loss＝lossgeneral+λ·lossanomaly(9)
fig. 10 is a schematic flow chart of a method for detecting goods according to an embodiment of the invention. The goods detection method operates in the container system 100 described above. The goods detection method comprises the following steps:
step S50, the detection model training device 50 generates a detection model having set model parameters;
step S51, the goods detection device 20 uses the detection model with the set model parameters to detect the goods of the collected goods image of the container 10 to generate a detection result;
step S52, the server 30 queries and updates the user data and the goods data according to the detection result;
further, the server 30 generates prompt information such as goods taking information, payment information, and abnormal information according to the detection result.
And step S53, the interactive terminal 40 interacts with the user according to the generated prompt message.
Specifically, the interactive terminal 40 implements basic interaction such as code scanning or user information acquisition.
In one embodiment, step S53 is specifically: the interactive terminal 40 controls the voice broadcaster to play prompt information; in another embodiment, step S53 is specifically: the interactive terminal 40 controls the display to play the prompt message; in another embodiment, step S53 is specifically: the interactive terminal 40 scans codes or recognizes human face to open the cargo cabinet door.
Referring to fig. 11, the implementation step of the step S51 specifically includes:
in step S511, the goods detection apparatus 20 performs multi-target detection on the goods image by using a multi-target detection algorithm (e.g., fast-RCNN, SSD) based on deep learning to obtain the deep visual characteristics and the prediction information of the target goods i.
Wherein i ∈ N+，I∈N+I is more than or equal to 1 and less than or equal to I, I is the number of the target goods, I is the total number of the target goods, N+Is a positive integer. The prediction information comprises first image data of a target item and first label information, and the first label information comprises class prediction cls of the target item iiAnd location box prediction bboxi. The first image data includes an image file name, and may further include information such as an image size and an image format.
Further, referring to fig. 12, the step S511 specifically includes:
step S5111, the goods detection device 20 maps the goods image to the depth visual feature of the location area where the target goods may exist;
in step S5112, the article detection device 20 predicts the class prediction cls of the target article i according to the depth visual featureiAnd location box prediction bboxi。
For example, when the fast-RCNN algorithm is employed, features in the goods image are extracted, and a plurality of region proposals (region offers) after being region-pooled are output, each region proposal including the depth visual feature. For example, when the SSD algorithm is used, a multi-scale feature map is obtained according to a product image, a feature included in each feature pixel point in the multi-scale feature map represents a plurality of candidate targets, the number of the candidate targets is the number of scale boxes (anchor boxes) of the multi-scale feature map, and each scale box includes the depth visual feature.
Step S512, the goods detection device obtains the abnormal prediction probability set P of the target goods i according to the depth visual characteristics by using a classification methodi＝{pi,k|k∈N+,K∈N+And K is more than or equal to 1 and less than or equal to K }.
And K is an abnormal type number, and K is the total number of abnormal types, wherein the abnormality comprises at least one of breakage, severe shielding and lodging.
Specifically, the article detection apparatus calculates an abnormality prediction probability set P of a target article i using equation (1)i(ii) a Where x is the input image, the function F represents the candidate object extractor, hiA feature representing an ith prediction target; function G represents the item Classification and predictive Box component, { clsi，bboxiIndicating the goods category and the position frame of which the ith prediction target is predicted; function H represents an anomaly classification module, PiA set of probabilities representing that the ith item is predicted to be in each anomaly category;
Figure BDA0002543431660000161
step S513, respectively judging the abnormal prediction probability sets PiThe anomaly prediction probability p of K anomalies in (1)i,kIf it is greater than the threshold t.
Step S514, the abnormal prediction probability pi,kAnd adding the abnormal information of the kth abnormality larger than the threshold value t into the label information to generate abnormal prediction information.
In this way, the determination of each abnormality type of the target item i and the generation of the abnormality prediction information are completed.
In particular, the classification method is a multi-label classification method.
For example, let K be 3, the abnormality types be 1. breakage, 2. severe occlusion, 3. lodging, and the set P of abnormality prediction probabilities of the target item 11If the threshold t is 0.5, the anomaly classification module 212 determines that there are 2 nd and 3 rd anomalies, i.e., there is a severe occlusion and a fall, for the target item 1. The corresponding abnormality detection information of the target goods 1 is {1.jpg, and the position frame predicts bbox1Class prediction cls1Severe occlusion, lodging }; as another example, assume the anomaly prediction probability set P for the target item 22If the threshold t is 0.5, then the target item 2 does not have any abnormality, and the detection information corresponding to the target item 2 is that{1.jpg, position box prediction bbox2Class prediction cls2}。
Referring to fig. 13, the implementation step of step S50 specifically includes:
step S501, the detection model training device 50 performs multi-target detection labeling on the training image to obtain a first labeled data set composed of labeled values of the target goods.
The labeling value comprises second image data and second label information of the target goods, and the second label information comprises category labeling and position frame labeling of the target goods.
The labeling format can be a Pascal VOC format, and the corresponding labeling tool can be labelImg; the annotation format may also be a YOLO format and the corresponding annotation tool may be YOLO _ mark. Other conventional formats and corresponding labeling tools are also possible. Through the labeling tools, a box is dragged for each target item by using a mouse to determine a position box and select a category, and then a label file in a corresponding format is saved and generated. For example, the first labeled data set for each target item is { image filename, location box label, category label }.
Step S502, the detection model training device 50 adds the abnormal label of the kth abnormality to the label value of each abnormal target item j in the first label data set to form an abnormal label value, so as to obtain a second label data set including the abnormal label value.
Wherein j ∈ N+，J∈N+J is more than or equal to 1 and less than or equal to J, J is the number of the target goods with the abnormal labels, J is the total number of the target goods with the abnormal labels, and gtboxjAnd marking the position frame of the jth target item with the abnormal mark.
For example, a training image 0001.jpg is subjected to multi-target detection annotation, the image comprises 3 target goods M1, M2 and M3, wherein abnormal annotations exist in the target goods M1 and M2, and no abnormal annotation exists in M3, and then the annotation data set of each target goods is as follows:
m1: {0001.jpg, position frame 1, item category 1, damaged, heavily occluded, lodging };
m2: {0001.jpg, position box 2, item category 2, severe occlusion, lodging };
m3: {0001.jpg, location box 3, item category 3 }.
In step S503, the detection model training device 50 calculates the abnormal classification loss function weighting coefficient λ using the following formula (2):
Figure BDA0002543431660000181
among them, a decimal number close to 0 is used. x is an image in the data set, and A is a set formed by images with abnormal labels in the second data set.
In practical situations, the number of abnormal target goods is much smaller than that of normal target goods, so that the problem of unbalanced category can be faced in the application scene of the invention, and the common solutions are as follows: oversampling, undersampling, cost sensitive learning, Focal local, and the like. The invention adopts an undersampling method to undersample the quantity of normal target goods with more quantity, divides a training set into A, B sets and calculates the value of lambda. Under-sampling methods are commonly used for classification tasks, but the scene of the invention is a detection task, namely, each image contains a plurality of targets and a plurality of categories, and the direct use of the under-sampling method has problems: in general, data acquisition is of a normal target, that is, goods in one image are normal and abnormal, but the goods in the set A may be abnormal, in this case, if the goods in one image is equal to 0, the abnormal classification module can never learn to distinguish the normal target, so the invention also adds a calculation formula of a 'cost-sensitive learning method' to solve the problem.
Specifically, the detection model training device 50 also detects a value calculated by the following equation (3):
Figure BDA0002543431660000191
wherein n is the total number of all target items in the second annotation data set, m1Number of target items in set A, m2As in set AThe number of normal target items.
In step S504, the detection model training device 50 calculates a joint loss function of the detection model according to the prediction information and the abnormal prediction information of the target item i generated by the item detection device 20, the second labeled data set, and the abnormal classification loss function weighting coefficient.
Step S505, the detection model training device 50 determines whether the joint loss function converges; if not, go to step S506; if yes, the process proceeds to step S507.
Step S506, the detection model training device 50 calculates the partial derivative of the current model parameter of the detection model based on the chain rule and the joint loss function, and subtracts the partial derivative from the current model parameter to obtain a new current model parameter; then, the process returns to step S504.
After returning to step S504, the goods detection device 20 generates corresponding prediction information and abnormal prediction information by using the new model parameters, and the detection model training device 50 performs a new joint loss function calculation according to the new prediction information and abnormal prediction information generated by the goods detection device 20, the second labeled data set, and the abnormal classification loss function weighting system.
In step S507, the detection model training device 50 finishes training, and the goods detection device 20 uses the new current model parameters for actual goods detection.
Referring to fig. 14, in step S504, the detection model training device 50 calculates a joint loss function of the detection model according to the forecast information and the abnormal forecast information of the target item i generated by the item detection device 20, the second labeled data set, and the abnormal classification loss function weighting coefficient, and specifically includes:
step S5041, calculating goods detection loss function lossgeneral；
In step S5042, an abnormal classification loss function loss is calculated by the following equations (4) to (8)anomaly；
Figure BDA0002543431660000192
Figure BDA0002543431660000201
Figure BDA0002543431660000202
Figure BDA0002543431660000203
Figure BDA0002543431660000204
Wherein L (i) represents a single item target hiA loss function of (d); iouiBbox representing the ith target item in the training imageiMaximum overlapping rate, y, with all target goods position frames with abnormal labelsi,kAn abnormal label showing whether the ith target item i has the k-th abnormality or not is given, if yi,kIf the value is 1, the abnormal label exists; if yi,kIf the value is 0, the abnormal label is not existed; gtboxjA position frame, bbox, indicating the jth target item with abnormal labeliA location box representing a predicted ith target item; gtclsj,kIndicating whether the k-th abnormal label of the jth abnormal label exists or not, idiAnd indicating the number of the target goods with the abnormal labels, which has the maximum overlapping rate with the ith target goods i.
Step S5043, calculating a joint loss function loss according to the goods detection function and the anomaly classification loss function by using the following formula (9):
loss＝lossgeneral+λ·lossanomaly(9)
the beneficial effects of the embodiment of the invention are as follows: different from the situation of the prior art, the container system and the goods detection method thereof in the embodiment of the invention can identify abnormal goods based on the pre-trained detection model and process the abnormal goods in time through the interactive terminal and the server.
In the embodiments provided in the present invention, the disclosed system, terminal and method can be implemented in other ways. For example, the above-described terminal/device/unit/module embodiments are illustrative, and the division of the unit/module is a logical function division, and there may be other division ways when the actual implementation is performed.
The units/modules described as separate parts may or may not be physically separate, i.e. may be located in one place, or may be distributed over a plurality of network elements. Some or all of the units/modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional units/modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
The integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the methods according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
It should be noted that the description of the present invention and the accompanying drawings illustrate preferred embodiments of the present invention, but the present invention may be embodied in many different forms and is not limited to the embodiments described in the present specification, which are provided as additional limitations to the present invention, and the present invention is provided for understanding the present disclosure more fully. Furthermore, the above-mentioned technical features are combined with each other to form various embodiments which are not listed above, and all of them are regarded as the scope of the present invention described in the specification; further, modifications and variations will occur to those skilled in the art in light of the foregoing description, and it is intended to cover all such modifications and variations as fall within the true spirit and scope of the invention as defined by the appended claims.

Claims (12)

1. The utility model provides a packing cupboard system, includes sales counter, goods detection device, server, mutual terminal, detection model trainer, goods detection device is arranged in gathering the goods image of goods in the sales counter, its characterized in that:
the detection model training device is used for training a detection model and generating the detection model with preset model parameters;
the goods detection device is also used for carrying out goods detection on the goods image by using the detection model with the preset model parameters to generate a detection result; wherein the article detection device includes a first processing unit including:
the goods detection module is used for carrying out multi-target detection on the goods image by adopting a multi-target detection algorithm based on deep learning to obtain the depth visual characteristics and the prediction information of the target goods; the prediction information comprises first image data and first label information of a target goods, and the first label information comprises category prediction and position frame prediction of the target goods;
an anomaly classification module to:
obtaining an abnormal prediction probability set P of the target goods according to the depth visual features by using a classification methodi＝{pi,k|k∈N+,K∈N+And K is more than or equal to 1 and less than or equal to K }; wherein K is the number of the abnormal category, K is the total number of the abnormal category, i is the number of the target goods, i belongs to N+，N+Is a positive integer;
respectively judging the abnormal prediction probability set PiThe anomaly prediction probability p of K anomalies in (1)i,kWhether greater than a threshold; and
predict the probability p of an anomalyi,kAnd adding the abnormal information of the kth abnormity which is larger than the threshold value into the label information to generate abnormity prediction information.
2. The container system of claim 1, wherein the detection model training apparatus comprises a storage unit for storing training images and a second processing unit, the second processing unit comprising:
the marking module is used for carrying out multi-target detection marking on the training image to obtain a first marking data set formed by marking values of target goods; the labeling value comprises second image data and second label information of the target goods, and the second label information comprises category labeling and position frame labeling of the target goods;
the abnormality labeling module is used for adding the abnormality label of the kth abnormality in the label value of each abnormal target goods in the first label data set to form an abnormality label value, so as to obtain a second label data set containing the abnormality label value;
a parameter training module comprising:
the loss function calculation submodule is used for calculating an abnormal classification loss function weighting coefficient; the system is also used for calculating a joint loss function of a detection model according to the forecast information of the target goods generated by the goods detection module, the abnormal forecast information generated by the abnormal classification module, the second labeled data set and the abnormal classification loss function weighting coefficient;
the model parameter setting submodule is used for calculating a partial derivative of the current model parameter of the detection model based on a chain rule and the combined loss function when the loss function calculating submodule judges that the combined loss function is not converged, and subtracting the partial derivative from the current model parameter to obtain a new current model parameter of the detection model; and the first processing unit is used for finishing the training when the loss function calculation submodule judges that the combined loss function is converged, and using the new current model parameters for actual goods detection.
3. The container system of claim 2, wherein the loss function computation submodule is configured to:
calculating an abnormal classification loss function weighting coefficient by using the formula (2); wherein λ is a decimal of 0, x is an image in the data set, and a is a set formed by images with abnormal labels in the second labeled data set;
Figure FDA0002543431650000021
calculating a goods detection loss function loss of the goods detection modulegeneral；
Calculating an abnormality classification loss function loss of the abnormality classification module using the formulas (4) to (8)anomaly(ii) a Wherein l (i) is used to calculate the loss of a single predicted target; iouiBbox representing ith target item in the training imageiThe maximum overlapping rate of the position frames of all the target goods with the abnormal labels is obtained; y isi,kAn abnormity label for indicating whether the ith target item has the k type abnormity, if yi,kIf the value is 1, the abnormal mark exists, and if yi,kIf the value is 0, the abnormal label is not existed; wherein i ∈ N+，I∈N+I is more than or equal to 1 and less than or equal to I, I is the number of the target goods, I is the total number of the target goods, N+Is a positive integer; gtboxjA position frame, bbox, indicating the jth target item with abnormal labeliA location box representing a predicted ith target item; gtclsj,kIndicating whether the k-th abnormal label of the jth abnormal label exists or not, idiIndicating the number of the target goods with the abnormal label with the maximum overlapping rate with the ith target goods i, wherein j is the number of the target goods with the abnormal label, and j belongs to N+；
Figure FDA0002543431650000031
Figure FDA0002543431650000032
Figure FDA0002543431650000033
Figure FDA0002543431650000034
Figure FDA0002543431650000035
According to the goods detection loss function lossgeneralThe anomaly classification loss function lossanomalyAnd the weighting coefficient lambda of the abnormal classification loss function, and the joint loss function loss is calculated by using the formula (9)
loss＝lossgeneral+λ·lossanomaly(9)。
4. The container system of claim 3, wherein the loss function calculation sub-module is further configured to utilize the value calculated by the equation (3):
Figure FDA0002543431650000036
wherein n is the total number of all target items in the second annotation data set, m1Number of anomalous target items in set A, m2The number of normal target items in the collection a.
5. The container system of claim 1, wherein the anomaly classification module is configured to calculate a set of anomaly prediction probabilities P for the target item using the equation (1)i(ii) a Where x is the input image, the function F represents the candidate object extractor, hiA feature representing an ith prediction target; the function G represents the goods classification and forecast box component,{clsi,bboxiexpressing the goods category and the position frame predicted by the ith prediction target; function H represents an anomaly classification module, PiSet of probabilities representing that ith item is predicted to be in respective exception categories
Figure FDA0002543431650000041
6. A goods detection method applied to the container system of any one of claims 1 to 5, wherein the method comprises the following steps:
the detection model training device generates a detection model with set model parameters;
the goods detection device utilizes the detection model with the set model parameters to carry out goods detection on the collected goods images of the goods in the container so as to generate a detection result; the method specifically comprises the following steps:
the goods detection device adopts a multi-target detection algorithm based on deep learning to carry out multi-target detection on the goods image to obtain the depth visual characteristics and the prediction information of the target goods; the prediction information comprises first image data and first label information of a target goods, and the first label information comprises category prediction and position frame prediction of the target goods;
the goods detection device obtains an abnormal prediction probability set P of the target goods according to the depth visual features by using a classification methodi＝{pi,k|k∈N+,K∈N+And K is more than or equal to 1 and less than or equal to K }; wherein K is the number of the abnormal category, K is the total number of the abnormal category, i is the number of the target goods, i belongs to N+，N+Is a positive integer;
the goods detection device respectively judges the abnormal prediction probability sets PiThe anomaly prediction probability p of K anomalies in (1)i,kWhether greater than a threshold;
the goods detecting apparatus predicts the abnormality probability pi,kAnd adding the abnormal information of the kth abnormity which is larger than the threshold value into the label information to generate abnormity prediction information.
7. The goods detection method according to claim 6, wherein the detection model training device generates the detection model with the set model parameters, and specifically comprises:
the detection model training device carries out multi-target detection and labeling on the training image to obtain a first labeling data set formed by labeling values of target goods; the labeling value comprises second image data and second label information of the target goods, and the second label information comprises category labeling and position frame labeling of the target goods;
the detection model training device adds the abnormal labels of the kth abnormality in the label value of each abnormal target goods j in the first label data set to form abnormal label values, and a second label data set comprising the abnormal labels is obtained;
the detection model training device sends the training image to the goods detection device, and prediction information and abnormal prediction information of the target goods i are generated through a detection model with set model parameters;
the detection model training device calculates an abnormal classification loss function weighting coefficient;
the detection model training device calculates a joint loss function of the detection model according to the prediction information and the abnormal prediction information of the target goods generated by the goods detection device, the second labeled data set and the abnormal classification loss function weighting coefficient;
when the detection model training device judges that the joint loss function is not converged, calculating a partial derivative of the current model parameter of the detection model based on a chain rule and the joint loss function, and subtracting the partial derivative from the current model parameter to obtain a new current model parameter of the detection model; and when the joint loss function is judged to be converged, finishing the training.
8. The method for detecting the good according to claim 7, wherein the training device for the detection model calculates the weighting coefficient of the abnormal classification loss function, and specifically comprises:
the detection model training device calculates an abnormal classification loss function weighting coefficient by using a formula (2); wherein λ is a decimal of 0, x is an image in the data set, and a is a set formed by images with abnormal labels in the second labeled data set;
Figure FDA0002543431650000051
the method for calculating the combined loss function of the detection model by the detection model training device according to the prediction information and the abnormal prediction information of the target goods generated by the goods detection device, the second labeled data set and the abnormal classification loss function weighting coefficient specifically comprises the following steps:
calculating the goods detection loss function lossgeneral；
Calculating an abnormal classification loss function loss using the formulas (4) to (8)anomaly(ii) a Wherein l (i) is used to calculate the loss of a single predicted target; iouiBbox representing ith target item in the training imageiThe maximum overlapping rate of the position frames of all the target goods with the abnormal labels is obtained; y isi,kAn abnormal label showing whether the ith target item i has the k-th abnormality or not is given, if yi,kIf the value is 1, the abnormal mark exists, and if yi,kIf the value is 0, the abnormal label is not existed; gtboxjA position frame, bbox, indicating the jth target item with abnormal labeliA location box representing a predicted ith target item; gtclsj,kIndicating whether the k-th abnormal label of the jth abnormal label exists or not, idiIndicating the number of the target goods with the abnormal label with the maximum overlapping rate with the ith target goods i, wherein j is the number of the target goods with the abnormal label, and j belongs to N+；
Figure FDA0002543431650000061
Figure FDA0002543431650000062
Figure FDA0002543431650000063
Figure FDA0002543431650000064
Figure FDA0002543431650000065
According to the goods detection loss function lossgeneralThe anomaly classification loss function lossanomalyAnd the weighting coefficient lambda of the abnormal classification loss function, and the joint loss function loss is calculated by using the formula (9)
loss＝lossgeneral+λ·lossanomaly(9)。
9. The method for detecting the good according to claim 8, wherein before the step of calculating the weighting coefficient of the abnormal classification loss function by the detection model training device using the formula (2), the method further comprises:
the detection model training means uses the value calculated by equation (3);
Figure FDA0002543431650000066
wherein n is the total number of all target items in the second annotation data set, m1Number of anomalous target items in set A, m2The number of normal target items in the collection a.
10. The item detection method of claim 6, wherein the item detection device uses a sorting method based on the depth visual featureObtaining an abnormal prediction probability set P of the target goodsi＝{pi,k|k∈N+,K∈N+And K is more than or equal to 1 and less than or equal to K, the method specifically comprises the following steps:
the goods detection device obtains an abnormal prediction probability set P of the target goods according to the depth visual characteristics by using a formula (1)i＝{pi,k|k∈N+,K∈N+And K is more than or equal to 1 and less than or equal to K }; where x is the input image, the function F represents the candidate object extractor, hiA feature representing an ith prediction target; function G represents the item Classification and predictive Box component, { clsi,bboxiIndicating the goods category and the position frame of which the ith prediction target is predicted; function H represents an anomaly classification module, PiSet of probabilities representing that ith item is predicted to be in respective exception categories
Figure FDA0002543431650000071
11. An article detection device, comprising a first processing unit, the first processing unit comprising:
the goods detection module is used for carrying out multi-target detection on the goods image by adopting a multi-target detection algorithm based on deep learning to obtain the depth visual characteristics and the prediction information of the target goods; the prediction information comprises first image data and first label information of a target goods, and the first label information comprises category prediction and position frame prediction of the target goods;
an anomaly classification module to:
obtaining an abnormal prediction probability set P of the target goods according to the depth visual features by using a classification methodi＝{pi,k|k∈N+,K∈N+And K is more than or equal to 1 and less than or equal to K }; wherein K is the number of the abnormal category, K is the total number of the abnormal category, i is the number of the target goods, i belongs to N+，N+Is a positive integer;
respectively judging the abnormal prediction probability set PiThe anomaly prediction probability p of K anomalies in (1)i,kWhether greater than a threshold; and
predict the probability p of an anomalyi,kAnd adding the abnormal information of the kth abnormity which is larger than the threshold value into the label information to generate abnormity prediction information.
12. An article detection method applied to the article detection device according to claim 11, characterized by comprising:
the goods detection device adopts a multi-target detection algorithm based on deep learning to carry out multi-target detection on the goods image to obtain the depth visual characteristics and the prediction information of the target goods; the prediction information comprises first image data and first label information of a target goods, and the first label information comprises category prediction and position frame prediction of the target goods;
the goods detection device obtains an abnormal prediction probability set P of the target goods according to the depth visual features by using a classification methodi＝{pi,k|k∈N+,K∈N+And K is more than or equal to 1 and less than or equal to K }; wherein K is the number of the abnormal category, K is the total number of the abnormal category, i is the number of the target goods, i belongs to N+，N+Is a positive integer;
the goods detection device respectively judges the abnormal prediction probability set PiThe anomaly prediction probability p of K anomalies in (1)i,kWhether greater than a threshold;
the goods detecting apparatus predicts the abnormality probability pi,kAnd adding the abnormal information of the kth abnormity which is larger than the threshold value into the label information to generate abnormity prediction information.
CN202010553634.5A 2020-06-17 2020-06-17 Container system, goods detection device and method Active CN111898417B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010553634.5A CN111898417B (en) 2020-06-17 2020-06-17 Container system, goods detection device and method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010553634.5A CN111898417B (en) 2020-06-17 2020-06-17 Container system, goods detection device and method

Publications (2)

Publication Number Publication Date
CN111898417A true CN111898417A (en) 2020-11-06
CN111898417B CN111898417B (en) 2023-08-08

Family

ID=73206740

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010553634.5A Active CN111898417B (en) 2020-06-17 2020-06-17 Container system, goods detection device and method

Country Status (1)

Country Link
CN (1) CN111898417B (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108038850A (en) * 2017-12-08 2018-05-15 天津大学 A kind of drainage pipeline Exception Type automatic testing method based on deep learning
CN108830183A (en) * 2018-05-28 2018-11-16 广东工业大学 A kind of the merchandise control method, apparatus and system of unmanned supermarket
CN108875831A (en) * 2018-06-22 2018-11-23 广州图匠数据科技有限公司 A kind of refrigerator-freezer commodity information identification method, system and device based on image recognition
CN109241946A (en) * 2018-10-11 2019-01-18 平安科技（深圳）有限公司 Abnormal behaviour monitoring method, device, computer equipment and storage medium
WO2020048492A1 (en) * 2018-09-05 2020-03-12 北京三快在线科技有限公司 Commodity state identification

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108038850A (en) * 2017-12-08 2018-05-15 天津大学 A kind of drainage pipeline Exception Type automatic testing method based on deep learning
CN108830183A (en) * 2018-05-28 2018-11-16 广东工业大学 A kind of the merchandise control method, apparatus and system of unmanned supermarket
CN108875831A (en) * 2018-06-22 2018-11-23 广州图匠数据科技有限公司 A kind of refrigerator-freezer commodity information identification method, system and device based on image recognition
WO2020048492A1 (en) * 2018-09-05 2020-03-12 北京三快在线科技有限公司 Commodity state identification
CN109241946A (en) * 2018-10-11 2019-01-18 平安科技（深圳）有限公司 Abnormal behaviour monitoring method, device, computer equipment and storage medium

Also Published As

Publication number Publication date
CN111898417B (en) 2023-08-08

Similar Documents

Publication Publication Date Title
CN108985359B (en) Commodity identification method, unmanned vending machine and computer-readable storage medium
CN109003390B (en) Commodity identification method, unmanned vending machine and computer-readable storage medium
CN110113226B (en) Method and device for detecting equipment abnormity
US9299229B2 (en) Detecting primitive events at checkout
US11669843B2 (en) Automated planogram anomaly detection
CN110570389A (en) Vehicle damage identification method and device
US20220391796A1 (en) System and Method for Mapping Risks in a Warehouse Environment
CN109191133B (en) Payment channel selection method and terminal equipment
CN111723777A (en) Method and device for judging commodity taking and placing process, intelligent container and readable storage medium
CN113269935B (en) Visual weighing method and weighing system based on screen-free weighing device
US11715290B2 (en) Machine learning based models for object recognition
CN110837824B (en) Commodity identification method for vending device, vending device and storage medium
CN112200196A (en) Phishing website detection method, device, equipment and computer readable storage medium
CN115034315A (en) Business processing method and device based on artificial intelligence, computer equipment and medium
CN113435900A (en) Transaction risk determination method and device and server
CN111898417B (en) Container system, goods detection device and method
CN116361488A (en) Method and device for mining risk object based on knowledge graph
CN116011822A (en) Store cashing management method and device, computer equipment and storage medium
CN115761457A (en) Commodity identification method and device, electronic equipment and computer readable medium
CN114255377A (en) Differential commodity detection and classification method for intelligent container
CN106851208A (en) A kind of approaches to IM and system
CN114842183A (en) Convolutional neural network-based switch state identification method and system
CN110472680B (en) Object classification method, device and computer-readable storage medium
CN109034067B (en) Method, system, equipment and storage medium for commodity image reproduction detection
CN112950329A (en) Commodity dynamic information generation method, device, equipment and computer readable medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant