CN111898417B

CN111898417B - Container system, goods detection device and method

Info

Publication number: CN111898417B
Application number: CN202010553634.5A
Authority: CN
Inventors: 钟华堡; 张帆
Original assignee: Xiamen Hualian Electronics Co Ltd
Current assignee: Xiamen Hualian Electronics Co Ltd
Priority date: 2020-06-17
Filing date: 2020-06-17
Publication date: 2023-08-08
Anticipated expiration: 2040-06-17
Also published as: CN111898417A

Abstract

The invention discloses a container system, a goods detection device and a method, wherein the method comprises the following steps: carrying out multi-target detection on the goods image by adopting a deep learning multi-target detection algorithm through the goods detection device to obtain deep visual characteristics and prediction information of target goods; obtaining an abnormal prediction probability set of the target goods according to the depth visual features by using a classification method; respectively judging abnormal prediction probability sets P _i Whether the abnormality prediction probability of K types of abnormalities in the database is larger than a threshold value; and adding the abnormal information of the kth abnormal with the abnormal prediction probability larger than the threshold value into the label information to generate the abnormal prediction information. According to the invention, the abnormal goods are identified based on the pre-trained detection model, and the abnormal goods are processed in time through the server and the interactive terminal.

Description

Container system, goods detection device and method

Technical Field

The embodiment of the invention relates to the technical field of image recognition, in particular to a container system, a goods detection device and a method.

Background

The existing visual unmanned sales counter adopts a dynamic image identification technology, and an order is counted and generated by identifying the dynamic in-out quantity and the category of goods at the entrance of the sales counter; or adopting a static image recognition technology to generate orders by recognizing and counting the goods quantity difference in the container before and after the container door is opened and closed.

In the existing image recognition technology, a multi-target detection technology (such as fast-RCNN and SSD) based on deep learning is generally adopted, and a target detection module is trained by utilizing a large amount of preset labeling image data, and can predict the position frame and the category of each target commodity in a commodity image of a sales counter for generating a shopping order of a user.

In practical application, when the goods are maliciously consumed by clients or the goods are unpacked from original packages, or key features of the goods are blocked or the goods fall down due to disordered arrangement of the goods, the image recognition technology cannot accurately recognize the goods. 1-3, if a broken article is mixed into a normal article, the existing method cannot be effectively distinguished, and the article can still be considered as an inventory article; as shown in fig. 4, whether the existing method can identify, and whether identification is required, for severely blocked goods is a controversial issue; as shown in fig. 5 and 6, the existing method cannot prompt the manager in the case of lodging of goods, and the situation of serious shielding is possibly changed at any time. In addition, a plurality of the above-described abnormal situations may occur at the same time, at which time it is not feasible to classify the abnormality using the above-described article detection technique.

Aiming at the problems, the prior goods detection technology can not effectively detect abnormal goods, so that the management of an unmanned sales counter is difficult, the risk of goods loss can not be ensured, and even serious food safety problems can be caused.

Disclosure of Invention

The technical problem which is mainly solved by the embodiment of the invention is to provide a container system and a goods detection method thereof, which can accurately identify and count the abnormal goods conditions in the sales counter so as to ensure the effective management of the goods of the unmanned sales counter.

In order to solve the technical problems, the invention adopts a technical scheme that: the goods detection device is used for collecting goods images of goods in the sales counter, and the detection model training device is used for training a detection model and generating the detection model with preset model parameters; the goods detection device is further used for detecting goods in the goods image by using the detection model with the preset model parameters so as to generate a detection result; wherein, goods detection device includes first processing unit, first processing unit includes: the goods detection module is used for carrying out multi-target detection on the goods image by adopting a multi-target detection algorithm based on deep learning to obtain deep visual characteristics and prediction information of target goods; the prediction information comprises first image data of a target article and first tag information, wherein the first tag information comprises category prediction and position frame prediction of the target article; the abnormality classification module is used for: obtaining an abnormal prediction probability set P of the target goods according to the depth visual features by using a classification method _i ＝{p _i,k |k∈N ⁺ ,K∈N ⁺ And K is more than or equal to 1 and less than or equal to K; wherein K is the number of abnormal categories, K is the total number of abnormal categories, and i is the target goodsNumbering, i.e. N ⁺ ，N ⁺ Is a positive integer; respectively judging the abnormal prediction probability set P _i Anomaly prediction probability p of K anomalies in a plurality _i,k Whether greater than a threshold; the anomaly prediction probability p _i,k And adding label information into the anomaly information of the kth anomaly which is larger than the threshold value to generate anomaly prediction information.

The detection model training device comprises a storage unit for storing training images and a second processing unit, wherein the second processing unit comprises: the labeling module is used for carrying out multi-target detection labeling on the training image to obtain a first labeling data set composed of labeling values of target goods; the labeling value comprises second image data of the target goods and second label information, wherein the second label information comprises category labels and position frame labels of the target goods; the anomaly labeling module is used for adding the anomaly labeling of the kth anomaly in the labeling value of each anomaly target goods in the first labeling data set to form an anomaly labeling value, and obtaining a second labeling data set containing the anomaly labeling value; a parameter training module, comprising: the loss function calculation sub-module is used for calculating an abnormal classification loss function weighting coefficient; the combined loss function of the detection model is calculated according to the prediction information of the target goods generated by the goods detection module, the abnormal prediction information generated by the abnormal classification module, the second marked data set and the abnormal classification loss function weighting coefficient; the model parameter setting submodule is used for calculating the partial derivative of the current model parameter of the detection model based on a chain rule and the joint loss function when the joint loss function is judged not to be converged by the loss function calculating submodule, and subtracting the partial derivative from the current model parameter to obtain a new current model parameter of the detection model; and the first processing unit is further used for ending training when the loss function calculation submodule judges that the joint loss function is converged, and the first processing unit uses new current model parameters for actual goods detection.

Wherein the loss function calculation submodule is used for: calculating an abnormal classification loss function weighting coefficient by using the formula (2); wherein λ is an anomaly classification loss function weighting coefficient, ε is a fraction of approximately 0, x is an image in the dataset, and A is a set of images in the second annotation dataset with anomaly annotations;

calculating a goods detection loss function loss of the goods detection module _general The method comprises the steps of carrying out a first treatment on the surface of the Calculating an abnormality classification loss function loss of the abnormality classification module using the formulas (4) to (8) _anomaly The method comprises the steps of carrying out a first treatment on the surface of the Wherein L (i) is used to calculate the loss of a single predicted target; iou (iou) _i Bbox representing the ith target item in the training image _i Maximum overlapping rate of the target goods position frames with the abnormal labels; y is _i,k An anomaly flag indicating whether the ith target article has the kth anomaly, if y _i,k If y is equal to 1, the anomaly flag is present _i,k =0, then the anomaly note is not present; wherein i is N ⁺ ，I∈N ⁺ I is not less than 1 and not more than I, I is the number of the target goods, I is the total number of the target goods, N ⁺ Is a positive integer; gtbox _j Position box of j-th target goods with abnormal labels, bbox _i A location box representing a predicted ith target item; gtcls _j,k The kth anomaly label of the target goods which indicates the jth anomaly label exists or not, and id _i The target item number with the abnormal label, which has the largest overlapping rate with the ith target item i, is represented, j is the target item number with the abnormal label, j epsilon N ⁺ ；

Detecting a loss function loss from the item _general The abnormal classification loss function loss _anomaly And the anomaly classification loss function weighting coefficient λ, calculating the joint loss function loss using equation (9)

loss＝loss _general +λ·loss _anomaly (9)。

Wherein the loss function calculation sub-module is further configured to calculate a value of ε using the formula (3):

wherein n is the total number of all target goods in the second marked data set, m ₁ For the number of abnormal target goods in the set A, m ₂ Is the number of normal target items in the collection a.

Wherein the anomaly classification module is used for calculating an anomaly prediction probability set P of the target goods by using the formula (1) _i The method comprises the steps of carrying out a first treatment on the surface of the Where x is the input image and function F represents the candidate object extractor, h _i Features representing the ith predicted target; function G represents the item classification and prediction box component, { cls _i ,bbox _i -representing the item category and location box predicted by the i < th > prediction target; function H represents an anomaly classification module, P _i Probability set representing that the ith item is predicted as each abnormal category

In order to solve the technical problems, the invention adopts another technical scheme that: there is provided a commodity detection method applied to a container system as described above, the method comprising: the detection model training device generates a detection model with set model parameters; the goods detection device detects goods in the collected goods images of the goods in the container by utilizing the detection model with the set model parameters so as to generate detection results; the method specifically comprises the following steps: the goods detection device carries out multi-target detection on the goods image by adopting a multi-target detection algorithm based on deep learning to obtain deep visual characteristics and prediction information of a target goods i; the prediction information comprises first image data of a target article and first tag information, wherein the first tag information comprises category prediction and position frame prediction of the target article; the goods detection device uses a classification method to obtain an abnormal prediction probability set P of the target goods according to the depth visual characteristics _i ＝{p _i,k |k∈N ⁺ ,K∈N ⁺ And K is more than or equal to 1 and less than or equal to K; wherein K is the number of abnormal categories, K is the total number of abnormal categories, i is the number of target goods, i is N ⁺ ，N ⁺ Is a positive integer; the goods detection device respectively judges the abnormal prediction probability set P _i Anomaly prediction probability p of K anomalies in a plurality _i,k Whether greater than a threshold; the article detecting device predicts the abnormality probability p _i,k And adding label information into the anomaly information of the kth anomaly which is larger than the threshold value to generate anomaly prediction information.

The detection model training device generates a detection model with set model parameters, and specifically comprises the following steps: the detection model training device carries out multi-target detection labeling on the training image to obtain a first labeling data set composed of labeling values of target goods; the labeling value comprises second image data of the target goods and second label information, wherein the second label information comprises category labels and position frame labels of the target goods; the detection model training device adds the abnormal label of the kth abnormality in the label value of each abnormal target goods j in the first label data set to form an abnormal label value, and a second label data set comprising the abnormal labels is obtained; the detection model training device sends the training image to the goods detection device, and generates prediction information and abnormal prediction information of the target goods i through a detection model with set model parameters; the detection model training device calculates an abnormal classification loss function weighting coefficient; the detection model training device calculates a joint loss function of the detection model according to the prediction information and the abnormal prediction information of the target goods generated by the goods detection device, the second labeling data set and the abnormal classification loss function weighting coefficient; when judging that the joint loss function is not converged, the detection model training device calculates the partial derivative of the current model parameter of the detection model based on a chain rule and the joint loss function, and subtracts the partial derivative from the current model parameter to obtain a new current model parameter of the detection model; and when the joint loss function is judged to be converged, ending training.

The method for calculating the abnormal classification loss function weighting coefficient by the detection model training device specifically comprises the following steps: the detection model training device calculates an abnormal classification loss function weighting coefficient by using the formula (2); wherein λ is an anomaly classification loss function weighting coefficient, ε is a fraction of approximately 0, x is a lower image in the dataset, and A is a set of images with anomaly annotations in the second annotation dataset;

the detection model training device calculates a joint loss function of a detection model according to the prediction information and the abnormal prediction information of the target goods generated by the goods detection device, the second labeling data set and the abnormal classification loss function weighting coefficient, and specifically includes: calculating a loss-of-good detection function loss _general The method comprises the steps of carrying out a first treatment on the surface of the Calculating an abnormality classification loss function loss of the abnormality classification module using the formulas (4) to (8) _anomaly The method comprises the steps of carrying out a first treatment on the surface of the Wherein L (i) is used to calculate a single predictionLoss of target; iou (iou) _i Bbox representing the ith target item in the training image _i Maximum overlapping rate of the target goods position frames with the abnormal labels; y is _i,k An anomaly flag indicating whether the kth anomaly exists in the ith target article i, if y _i,k If y is equal to 1, the anomaly flag is present _i,k =0, then the anomaly note is not present; gtbox _j Position frame, bbox, of jth target goods with abnormal labels _i A location box representing a predicted ith target item; gtcls _j,k The kth anomaly label of the target goods which indicates the jth anomaly label exists or not, and id _i The target item number with the abnormal label, which has the largest overlapping rate with the ith target item i, is represented, j is the target item number with the abnormal label, j epsilon N ⁺ ；

loss＝loss _general +λ·loss _anomaly (9)。

Wherein before the "the detection model training apparatus calculates the abnormal classification loss function weighting coefficient" by using the formula (2), the method further includes: the detection model training device calculates the value of epsilon by using the formula (3);

Wherein the article detecting device obtains the abnormal prediction probability set P of the target article according to the depth visual features by using a classification method _i ＝{p _i,k |k∈N ⁺ ,K∈N ⁺ And K is more than or equal to 1 and less than or equal to K } ", specifically comprising: the article detection device obtains an abnormal prediction probability set P of the target article according to the depth visual characteristics by using a formula (1) _i ＝{p _i,k |k∈N ⁺ ,K∈N ⁺ And K is more than or equal to 1 and less than or equal to K; where x is the input image and function F represents the candidate object extractor, h _i Features representing the ith predicted target; function G represents the item classification and prediction box component, { cls _i ,bbox _i -representing the item category and location box for which the i-th prediction target is predicted; function H represents an anomaly classification module, P _i Probability set representing that the ith item is predicted as each abnormal category

In order to solve the technical problems, the invention adopts another technical scheme that: there is provided an article detecting apparatus including a first processing unit including: the goods detection module is used for carrying out multi-target on the goods image by adopting a multi-target detection algorithm based on deep learningDetecting to obtain depth visual characteristics and prediction information of a target goods; the prediction information comprises first image data of a target article i and first tag information, wherein the first tag information comprises category prediction and position frame prediction of the target article; the abnormality classification module is used for: obtaining an abnormal prediction probability set P of the target goods according to the depth visual features by using a classification method _i ＝{p _i,k |k∈N ⁺ ,K∈N ⁺ And K is more than or equal to 1 and less than or equal to K; wherein K is the number of abnormal categories, K is the total number of abnormal categories, i is the number of target goods, i is N ⁺ ，N ⁺ Is a positive integer; respectively judging the abnormal prediction probability set P _i Anomaly prediction probability p of K anomalies in a plurality _i,k Whether greater than a threshold; the anomaly prediction probability p _i,k And adding label information into the anomaly information of the kth anomaly which is larger than the threshold value to generate anomaly prediction information.

In order to solve the technical problems, the invention adopts a technical scheme that: there is provided an article detecting method, applied to the article detecting apparatus as described above, the method comprising: the goods detection device carries out multi-target detection on the goods image by adopting a multi-target detection algorithm based on deep learning to obtain deep visual characteristics and prediction information of target goods; the prediction information comprises first image data of a target article i and first tag information, wherein the first tag information comprises category prediction and position frame prediction of the target article i; the goods detection device uses a classification method to obtain an abnormal prediction probability set P of the target goods according to the depth visual characteristics _i ＝{p _i,k |k∈N ⁺ ,K∈N ⁺ And K is more than or equal to 1 and less than or equal to K; wherein K is the number of abnormal categories, K is the total number of abnormal categories, i is the number of target goods, i is N ⁺ ，N ⁺ Is a positive integer; the goods detection device respectively judges the abnormal prediction probability set P _i Anomaly prediction probability p of K anomalies in a plurality _i,k Whether greater than a threshold; the article detecting device predicts the abnormality probability p _i,k And adding label information into the anomaly information of the kth anomaly which is larger than the threshold value to generate anomaly prediction information.

The beneficial effects of the embodiment of the invention are as follows: different from the situation of the prior art, the container system and the goods detection method thereof in the embodiment of the invention can identify the abnormal goods based on the pre-trained detection model and timely process the abnormal goods through the interactive terminal and the server.

Drawings

FIG. 1 is a first image schematic view of an article in the event of breakage;

FIG. 2 is a second image schematic diagram of an article in the event of breakage;

FIG. 3 is a third image schematic diagram of an article in the event of breakage;

FIG. 4 is a schematic image of an article with severe occlusion;

FIG. 5 is a schematic image of an article in the event that the article is severely occluded and the article is lodged;

FIG. 6 is a schematic image of an item in a lodging condition;

FIG. 7 is a schematic diagram of a sales counter system in accordance with an embodiment of the present invention;

fig. 8 is a functional block diagram of an article identification apparatus according to a first embodiment of the present invention;

FIG. 9 is a functional block diagram of a test model training device in an embodiment of the invention;

FIG. 10 is a flow chart of a method of detecting an item in an embodiment of the present invention;

FIG. 11 is a flow chart of the implementation method of step S2 shown in FIG. 10;

FIG. 12 is a flow chart of the implementation method of step S22 shown in FIG. 11;

FIG. 13 is a flow chart of the implementation method of step S1 shown in FIG. 10;

fig. 14 is a flowchart of the implementation method of step S14 shown in fig. 13.

Detailed Description

In order that the invention may be readily understood, a more particular description thereof will be rendered by reference to specific embodiments that are illustrated in the appended drawings.

In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 7 is a schematic diagram of a container system 100 according to an embodiment of the invention. The container system 100 includes a sales counter 10, a commodity detection device 20, a server 30 storing user data and commodity data, an interactive terminal 40, and a detection model training device 50. The article detecting device 20 is communicatively connected to the server 30 and the interactive terminal 40, respectively.

The test model training device 50 is used for training a test model and generating a test model with preset model parameters.

The goods detection device 20 is configured to collect goods images of goods in a sales counter, perform goods detection on the goods images of the counter 10 by using the detection model with preset model parameters to generate detection results, and perform goods detection on input goods training images based on the detection model parameters to generate detection results, so as to assist in completing detection model training.

The server 30 is configured to query and update the user data and the product data according to the detection result. Further, the server 30 is further configured to generate prompt information such as goods taking information, payment information, and abnormal information according to the detection result, and send the prompt information to the interaction terminal 40. Specifically, in one embodiment, when the user frequently causes an abnormal prompt, the server 40 decreases the credit score of the user according to the abnormal prompt information.

The interactive terminal 40 is used for interacting with a user according to the prompt information generated by the server 30. Further, the interaction terminal 40 is further configured to implement basic interaction such as code scanning or user information collection. Specifically, in an embodiment, the interactive terminal 40 controls the voice broadcaster to play the prompt information; in another embodiment, the interactive terminal 40 controls the display to play the prompt message; in yet another embodiment, the interactive terminal 40 performs code scanning or face scanning recognition to open the cargo door.

Referring to fig. 8, a schematic diagram of a device 20 for detecting goods according to an embodiment of the present invention is shown. The article detecting device 20 comprises a first processing unit 21 and an image acquisition unit 22 connected with the first processing unit 21, wherein the image acquisition unit 22 is used for acquiring an article image of an article in the container 10. The first processing unit 21 includes a commodity detection module 211 and an abnormality classification module 212.

The item detection module 211 is configured to perform multi-target detection on the item image by using a multi-target detection algorithm (such as fast-RCNN, SSD) based on deep learning, to obtain a deep visual feature and prediction information of a target item i, where i e N ⁺ ，I∈N ⁺ I is not less than 1 and not more than I, I is the number of the target goods, I is the total number of the target goods, N ⁺ Is a positive integer. The prediction information comprises first image data of the target goods i and first label information, and the first label information comprises category prediction cls of the target goods i _i And position box prediction bbox _i . The first image data includes an image file name, and may further include information such as an image size and an image format.

Specifically, the item detection module 211 includes a candidate target extractor 2111, an item classification and prediction box component 2112. The candidate target extractor 2111 is used to map an item image to a depth visual feature of a location area where a target item may exist. The item classification and prediction box component 2112 is for predicting a category prediction cls for a target item i based on the depth visual features _i And position box prediction bbox _i 。

For example, when the fast-RCNN algorithm is used, the candidate object extractor 221 is configured to extract features in the item image and output a plurality of region proposals (regional propositions) after being region-pooled, each region proposal including the depth visual feature. For example, when the SSD algorithm is adopted, the candidate object extractor 221 is configured to obtain a multi-scale feature map according to the item image, where each feature pixel in the multi-scale feature map includes features that represent multiple candidate objects, and the number of candidate objects is the number of scale frames (frames) of the multi-scale feature map, and each scale frame includes the depth visual feature.

The anomaly classification module 212 is configured to:

obtaining an abnormal prediction probability set P of the target goods i according to the depth visual features by using a classification method _i ＝{p _i,k |k∈N ⁺ ,K∈N ⁺ And K is more than or equal to 1 and less than or equal to K; wherein K is the number of abnormal species, and K is the total number of abnormal species; the abnormality includes at least one of breakage, severe occlusion, lodging;

judging abnormality prediction probability p of kth abnormality _i,k Whether greater than a threshold t; and

in determining the anomaly prediction probability p of the kth anomaly _i,k When the value is larger than a threshold t, determining that the k-th abnormality exists in the target goods i, and adding the abnormality information of the k-th abnormality into the tag information to generate abnormality prediction information. Specifically, the classification method is a multi-label classification method.

Specifically, the anomaly classification module 212 calculates a set of anomaly prediction probabilities P for the target good i using the equation (1) _i The method comprises the steps of carrying out a first treatment on the surface of the Where x is the input image and function F represents the candidate object extractor, h _i Features representing the ith predicted target; function G represents the item classification and prediction box component, { cls _i ，bbox _i -representing the item category and location box predicted by the i < th > prediction target; function H represents an anomaly classification module, P _i A set of probabilities representing that the ith item is predicted to be of each abnormal category;

for example, let k=3, the abnormality types are in order of 1. Breakage, 2. Serious occlusion, 3. Lodging, abnormality prediction probability set P for target article 1 ₁ = {0.1,0.6,0.9}, with a threshold t=0.5, the anomaly classification module 212 determines that the target good 1 has anomalies of type 2 and 3, i.e., there is a severe occlusionAnd lodging. The abnormality detection information corresponding to the target article 1 is {1.Jpg, bbox is predicted by a position frame ₁ Class prediction cls ₁ Severe occlusion, lodging }; in another example, the anomaly prediction probability set P of the target article 2 is set ₂ = {0.1,0.2,0.2}, the threshold t=0.5, then no abnormality exists in the target article 2, the detection information corresponding to the target article 2 is {1.Jpg, and the position frame prediction bbox ₂ Class prediction cls ₂ }。

Fig. 9 is a schematic structural diagram of a test model training device according to an embodiment of the invention. The detection model training device 50 comprises a second processing unit 51 and a storage unit 52, wherein the storage unit 52 is used for storing training images. The second processing unit 51 includes a labeling module 511, an anomaly labeling module 512, and a parameter training module 513.

The labeling module 511 is configured to perform multi-target detection labeling on the training image, so as to obtain a first labeling data set composed of labeling values of the target goods i; the labeling value comprises second image data of the target goods i and second label information, and the second label information comprises category labels and position frame labels of the target goods i. The labeling format can be a Pascal VOC format, and the corresponding labeling tool can be labelImg; the annotation format may also be YOLO format and the corresponding annotation tool may be yolo_mark. Other conventional formats and corresponding labeling tools are also possible. Through the labeling tools, a box is dragged by a mouse for each target goods to determine a position box and select a category, and then a label file in a corresponding format is saved and generated. For example, the first labeling data set of each target item is { image file name, location box label, category label }.

The anomaly labeling module 512 is configured to add an anomaly label of the kth anomaly to the labeling value of each anomaly target item j in the first labeling data set to form an anomaly labeling value, and obtain a second labeling data set containing the anomaly labeling value; wherein j is E N ⁺ ，J∈N ⁺ J is not less than 1 and not more than J, J is the number of the target goods with abnormal labels, J is the total number of the target goods with abnormal labels, and gtbox _j For j-th presence of heteroAnd labeling the position frame of the commonly labeled target goods.

For example, performing in-process multi-target detection labeling on one training image 0001.Jpg, wherein the image comprises 3 target goods M1, M2 and M3, and the target goods M1 and M2 are labeled abnormally, and the labeling data set of each target goods is as follows:

m1: {0001.Jpg, position frame 1, item category 1, broken, severely occluded, lodging };

m2: {0001.Jpg, position box 2, item category 2, severe occlusion, lodging };

m3: {0001.Jpg, location box 3, item category 3}.

The parameter training module 513 includes:

a loss function calculation submodule 5131 for calculating an abnormal classification loss function weighting coefficient lambda; and the method is further used for calculating a joint loss function of a detection model according to the prediction information of the target goods i generated by the goods detection module 211, the abnormal prediction information generated by the abnormal classification module 212, the second marked data set generated by the abnormal marking module 512 and the abnormal classification loss function weighting coefficient lambda, and judging whether the joint loss function is converged.

A model parameter setting submodule 5132, configured to calculate a partial derivative of a current model parameter of the detection model based on a chained rule and the joint loss function when the loss function calculation submodule 5132 determines that the joint loss function is not converged, and subtract the partial derivative from the current model parameter to obtain a new current model parameter of the detection model; and is further configured to end training when the loss function calculation submodule 5132 determines that the joint loss function converges.

Further, the loss function calculation sub-module 5131 is configured to calculate a good detection loss function of the good detection module 211 and an abnormal classification loss function of the abnormal classification module 212, and calculate the joint loss function according to the good detection loss function and the abnormal classification loss function.

Specifically, the loss function calculation sub-module 5132 calculates the anomaly classification loss function weighting coefficient λ using the following formula (2):

where ε is a fraction of approximately 0. x is an image under the data set, and A is a set formed by images with abnormal labels in the second label data set.

In the actual situation, the number of abnormal target goods is much smaller than that of normal target goods, so the problem of unbalanced categories is faced in the application scene of the invention, and the common solution method is as follows: oversampling, undersampling, cost sensitive learning, focal Loss, and the like. The invention adopts an undersampling method to undersample the number of normal target goods with relatively large numbers, divides a training set into A, B sets, and calculates the value of lambda. The undersampling method is commonly used for classifying tasks, but the scene of the invention is a detection task, namely, each image contains a plurality of targets and various categories, and the direct use of the undersampling method has problems: in general, the data collection is of a normal target, namely, goods in one image are normal and abnormal, but the situation that the goods in the set A are possibly all abnormal exists, in this case, if epsilon is equal to 0, the abnormal classification module can never learn to distinguish the normal target, so the invention also adds a calculation formula of epsilon' (cost sensitive learning method) to solve the problem.

Specifically, the loss function calculation sub-module 5132 also calculates the value of ε using the following formula (3):

The loss function calculation submodule 5132 calculates a good detection loss function loss of the good detection module _general ，loss _general Is defined by a specific multi-objective detection algorithm (such as Faster-RCNN, SSD);

the loss function calculation submodule 5132 calculates an abnormality classification loss function loss of the abnormality classification module 212 using the following formulas (4) to (8) _anomaly ：

Wherein L (i) represents a single article target h _i Is a loss function of iou _i Bbox representing the ith target item in the training image _i Maximum overlapping rate, y of target goods position frames with abnormal labels _i，k An anomaly flag indicating whether the kth anomaly exists in the ith target article i, if y _k =1, then this anomaly note exists; if y _k And=0, then this anomaly note does not exist. gtbox _i Position frame, bbox, of jth target goods with abnormal labels _i A location box representing a predicted ith target item; gtcls _j,k The kth anomaly label of the target goods which indicates the jth anomaly label exists or not, and id _i And the target item number with the abnormal label with the largest overlapping rate with the ith target item i is represented.

Further, the loss function calculation sub-module 5132 calculates the joint loss function loss using the following formula (9):

loss＝loss _general +λ·loss _anomaly (9)

referring to fig. 10, a flow chart of a method for detecting an article according to an embodiment of the invention is shown. The item detection method operates in the container system 100 described above. The goods detection method comprises the following steps:

step S50, the detection model training device 50 generates a detection model with set model parameters;

step S51, the goods detection device 20 adopts the detection model with set model parameters to carry out goods detection on the collected goods image of the container 10 so as to generate a detection result;

step S52, the server 30 queries and updates the user data and the goods data according to the detection result;

further, the server 30 also generates prompt information such as goods taking information, payment information, abnormal information, etc. according to the detection result.

In step S53, the interactive terminal 40 interacts with the user according to the generated prompt information.

Specifically, the interaction terminal 40 implements basic interaction such as code scanning or user information collection.

In one embodiment, step S53 is specifically: the interactive terminal 40 controls the voice broadcasting device to play the prompt information; in another embodiment, step S53 is specifically: the interactive terminal 40 controls the display to play the prompt information; in yet another embodiment, step S53 is specifically: the interactive terminal 40 performs code scanning or face scanning recognition to open the cargo door.

Referring to fig. 11, the implementation step of step S51 specifically includes:

in step S511, the article detection device 20 performs multi-objective detection on the article image by using a multi-objective detection algorithm (e.g. fast-RCNN, SSD) based on deep learning, so as to obtain the deep visual features and the prediction information of the target article i.

Wherein i is N ⁺ ，I∈N ⁺ I is not less than 1 and not more than I, I is the number of the target goods, I is the total number of the target goods, N ⁺ Is a positive integer. The prediction information comprises first image data of the target goods and first label information, wherein the first label information comprises category prediction cls of the target goods i _i And position box prediction bbox _i . The first image data includes an image file name, and may further include information such as an image size and an image format.

Further, referring to fig. 12, the step S511 specifically includes:

step S5111, the article detecting apparatus 20 maps the article image to a depth visual feature of a location area where the target article may exist;

Step S5112, the article detecting apparatus 20 predicts the category prediction cls of the target article i based on the depth visual feature _i And position box prediction bbox _i 。

For example, when the fast-RCNN algorithm is employed, features in the item image are extracted, and a plurality of region proposals (region propositions) after being region-pooled, each region proposal including the depth visual feature, are output. For example, when an SSD algorithm is adopted, a multi-scale feature map is obtained according to an article image, features contained in each feature pixel point in the multi-scale feature map represent a plurality of candidate targets, wherein the number of candidate targets is the number of scale frames (anchors) of the multi-scale feature map, and each scale frame includes the depth visual feature.

Step S512, the article detection device obtains an abnormal prediction probability set P of the target article i according to the depth visual features by using a classification method _i ＝{p _i,k |k∈N ⁺ ,K∈N ⁺ And K is more than or equal to 1 and less than or equal to K.

K is the number of abnormal types, and K is the total number of abnormal types, wherein the abnormality comprises at least one of breakage, serious shielding and lodging.

Specifically, the article detecting device calculates the abnormal prediction probability set P of the target article i using the formula (1) _i The method comprises the steps of carrying out a first treatment on the surface of the Where x is the input image and function F represents the candidate object extractor, h _i Representation ofCharacteristics of the ith predicted target; function G represents the item classification and prediction box component, { cls _i ，bbox _i -representing the item category and location box for which the i-th prediction target is predicted; function H represents an anomaly classification module, P _i A set of probabilities representing that the ith item is predicted to be of each abnormal category;

step S513 of judging the anomaly prediction probability sets P _i Anomaly prediction probability p of K anomalies in a plurality _i,k Whether greater than a threshold t.

Step S514, predicting the anomaly probability p _i,k And adding label information into the anomaly information of the kth anomaly greater than the threshold t to generate anomaly prediction information.

Thus, the judgment of each abnormality type of the target article i and the generation of abnormality prediction information are completed.

Specifically, the classification method is a multi-label classification method.

For example, let k=3, the abnormality types are in order of 1. Breakage, 2. Serious occlusion, 3. Lodging, abnormality prediction probability set P for target article 1 ₁ {0.1,0.6,0.9}, the threshold t=0.5, the anomaly classification module 212 determines that the target good 1 has anomalies 2 and 3, i.e., severe occlusion and lodging. The abnormality detection information corresponding to the target article 1 is {1.Jpg, bbox is predicted by a position frame ₁ Class prediction cls ₁ Severe occlusion, lodging }; in another example, the anomaly prediction probability set P of the target article 2 is set ₂ = {0.1,0.2,0.2}, the threshold t=0.5, then no abnormality exists in the target article 2, the detection information corresponding to the target article 2 is {1.Jpg, and the position frame prediction bbox ₂ Class prediction cls ₂ }。

Referring to fig. 13, the implementation step of step S50 specifically includes:

in step S501, the detection model training apparatus 50 performs multi-target detection labeling on the training image to obtain a first labeling data set composed of labeling values of target goods.

The labeling value comprises second image data of the target goods and second label information, and the second label information comprises category labels and position frame labels of the target goods.

The labeling format can be a Pascal VOC format, and the corresponding labeling tool can be labelImg; the annotation format may also be YOLO format and the corresponding annotation tool may be yolo_mark. Other conventional formats and corresponding labeling tools are also possible. Through the labeling tools, a box is dragged by a mouse for each target goods to determine a position box and select a category, and then a label file in a corresponding format is saved and generated. For example, the first labeling data set of each target item is { image file name, location box label, category label }.

In step S502, the detection model training apparatus 50 adds the abnormal label of the kth abnormality to the label value of each abnormal target article j in the first label data set to form an abnormal label value, and obtains a second label data set including the abnormal label value.

Wherein j is E N ⁺ ，J∈N ⁺ J is not less than 1 and not more than J, J is the number of the target goods with abnormal labels, J is the total number of the target goods with abnormal labels, and gtbox _j And marking the position frame of the j-th target goods with abnormal marks.

m2: {0001.Jpg, position box 2, item category 2, severe occlusion, lodging };

m3: {0001.Jpg, location box 3, item category 3}.

In step S503, the detection model training apparatus 50 calculates the abnormal classification loss function weighting coefficient λ using the following formula (2):

where ε is a fraction of approximately 0. x is an image under the data set, and A is a set formed by images with abnormal labels in the second data set.

Specifically, the detection model training apparatus 50 also calculates the value of ε using the following formula (3):

wherein n is the total number of all target goods in the second marked data set, m ₁ For the number of target items in set A, m ₂ Is the number of normal target items in the collection a.

In step S504, the detection model training device 50 calculates a joint loss function of the detection model based on the prediction information and the abnormal prediction information of the target item i generated by the item detection device 20, the second labeled data set, and the abnormal classification loss function weighting coefficient.

Step S505, the test model training apparatus 50 determines whether the joint loss function converges; if not, go to step S506; if yes, the process proceeds to step S507.

Step S506, the detection model training device 50 calculates the partial derivative of the current model parameter of the detection model based on the chain rule and the joint loss function, and subtracts the partial derivative from the current model parameter to obtain a new current model parameter; then, the process returns to step S504.

After returning to step S504, the article inspection device 20 generates corresponding prediction information and abnormal prediction information by using the new model parameters, and the inspection model training device 50 calculates a new joint loss function according to the new prediction information and abnormal prediction information generated by the article inspection device 20, the second labeling data set, and the abnormal classification loss function weighting system.

In step S507, the detection model training device 50 ends the training, and the item detection device 20 uses the new current model parameters for actual item detection.

Referring to fig. 14, in step S504, the test model training device 50 calculates a joint loss function of the test model according to the prediction information and the abnormal prediction information of the target item i generated by the item detection device 20, the second labeling data set, and the abnormal classification loss function weighting coefficient, which specifically includes:

step S5041, calculating a good detection loss function loss _general ；

Step S5042, calculating an abnormal classification loss function loss using the following formulas (4) to (8) _anomaly ；

Wherein L (i) represents a single article target h _i A loss function of (2); iou (iou) _i Bbox representing the ith target item in the training image _i Maximum overlapping rate, y of target goods position frames with abnormal labels _i,k An anomaly flag indicating whether the kth anomaly exists in the ith target article i, if y _i,k =1, then this anomaly note exists; if y _i,k =0, then this anomaly note does not exist; gtbox _j Position frame, bbox, of jth target goods with abnormal labels _i A location box representing a predicted ith target item; gtcls _j,k The kth anomaly label of the target goods which indicates the jth anomaly label exists or not, and id _i And the target item number with the abnormal label with the largest overlapping rate with the ith target item i is represented.

Step S5043, calculating a joint loss function loss according to the item detection function and the abnormal classification loss function by using the following formula (9):

loss＝loss _general +λ·loss _anomaly (9)

In the embodiments provided in the present invention, the disclosed system, terminal and method may be implemented in other manners. For example, the terminal/device/unit/module embodiments described above are illustrative, and the unit/module division is a logic function division, and there may be other division manners in actual implementation.

The units/modules described as separate components may or may not be physically separate, i.e. may be located in one place, or may be distributed over a plurality of network elements. Some or all of the units/modules may be selected according to actual needs to achieve the purpose of the embodiment.

In addition, each functional unit/module in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in hardware plus software functional units.

The integrated units implemented in the form of software functional units described above may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium, and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform part of the steps of the methods according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

It should be noted that the description of the present invention and the accompanying drawings illustrate preferred embodiments of the present invention, but the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, which are not to be construed as additional limitations of the invention, but are provided for a more thorough understanding of the present invention. The above-described features are continuously combined with each other to form various embodiments not listed above, and are considered to be the scope of the present invention described in the specification; further, modifications and variations of the present invention may be apparent to those skilled in the art in light of the foregoing teachings, and all such modifications and variations are intended to be included within the scope of this invention as defined in the appended claims.

Claims

1. The utility model provides a packing cupboard system, includes sales counter, goods detection device, server, interactive terminal, detects model trainer, goods detection device is used for gathering the goods image of goods in the sales counter, its characterized in that:

the detection model training device is used for training a detection model and generating the detection model with preset model parameters;

the goods detection device is further used for detecting goods in the goods image by using the detection model with the preset model parameters so as to generate a detection result; wherein, goods detection device includes first processing unit, first processing unit includes:

the goods detection module is used for carrying out multi-target detection on the goods image by adopting a multi-target detection algorithm based on deep learning to obtain deep visual characteristics and prediction information of target goods; the prediction information comprises first image data of a target article and first tag information, wherein the first tag information comprises category prediction and position frame prediction of the target article;

the abnormality classification module is used for:

obtaining an abnormal prediction probability set P of the target goods according to the depth visual features by using a classification method _i ＝{p _i,k |k∈N ⁺ ,K∈N ⁺ And K is more than or equal to 1 and less than or equal to K; wherein K is the number of abnormal categories, K is the total number of abnormal categories, i is the number of target goods, i is N ⁺ ，N ⁺ Is a positive integer;

respectively judging the abnormal prediction probability set P _i Anomaly prediction probability p of K anomalies in a plurality _i,k Whether greater than a threshold; and

prediction probability p of abnormality _i,k Adding label information into the abnormal information of the kth abnormal which is larger than the threshold value to generate abnormal prediction information;

the detection model training device comprises a storage unit and a second processing unit, wherein the storage unit is used for storing training images; the second processing unit includes:

the labeling module is used for carrying out multi-target detection labeling on the training image to obtain a first labeling data set composed of labeling values of target goods; the labeling value comprises second image data of the target goods and second label information, wherein the second label information comprises category labels and position frame labels of the target goods;

the anomaly labeling module is used for adding the anomaly labeling of the kth anomaly in the labeling value of each anomaly target goods in the first labeling data set to form an anomaly labeling value, and obtaining a second labeling data set containing the anomaly labeling value; and

a parameter training module, comprising:

A loss function calculation sub-module for:

calculating an abnormal classification loss function weighting coefficient by using a formula (2); wherein λ is an anomaly classification loss function weighting coefficient, ε is a fraction of approximately 0, x is an image in the dataset, and A is a set of images in the second annotation dataset with anomaly annotations;

calculating the value of ε using equation (3); wherein n is the total number of all target goods in the second marked data set, m ₁ For the number of abnormal target goods in the set A, m ₂ The number of normal target goods in the set A;

calculating a goods detection loss function loss of the goods detection module _general ；

Calculating an anomaly classification loss function loss of the anomaly classification module by using formulas (4) to (8) _anomaly The method comprises the steps of carrying out a first treatment on the surface of the Wherein L (i) is used for a meterCalculating the loss of a single prediction target; iou (iou) _i Bbox representing the ith target item in the training image _i Maximum overlapping rate of the target goods position frames with the abnormal labels; y is _i,k An anomaly flag indicating whether the ith target article has the kth anomaly, if y _i,k If y is equal to 1, the anomaly flag is present _i,k =0, then the anomaly note is not present; wherein i is N ⁺ ，I∈N ⁺ I is not less than 1 and not more than I, I is the number of the target goods, I is the total number of the target goods, N ⁺ Is a positive integer; gtbox _j Position frame, bbox, of jth target goods with abnormal labels _i A location box representing a predicted ith target item; gtcls _j,k The kth anomaly label of the target goods which indicates the jth anomaly label exists or not, and id _i The target item number with the abnormal label, which has the largest overlapping rate with the ith target item i, is represented, j is the target item number with the abnormal label, j epsilon N ⁺ ；

Detecting a loss function loss from the item _general The abnormal classification loss function loss _anomaly And the anomaly classification loss function weighting coefficient lambda, calculating a joint loss function loss using equation (9);

loss＝loss _general +λ·loss _anomaly (9)

the model parameter setting submodule is used for calculating the partial derivative of the current model parameter of the detection model based on a chain rule and the joint loss function when the joint loss function is judged not to be converged by the loss function calculating submodule, and subtracting the partial derivative from the current model parameter to obtain a new current model parameter of the detection model; and the first processing unit is further used for ending training when the loss function calculation submodule judges that the joint loss function is converged, and the first processing unit uses new current model parameters for actual goods detection.

2. The container system of claim 1, wherein the anomaly classification module is configured to calculate the anomaly prediction probability set P for the target good using equation (1) _i The method comprises the steps of carrying out a first treatment on the surface of the Where x is the input image and function F represents the candidate object extractor, h _i Features representing the ith predicted target; function G represents the item classification and prediction box component, { cls _i ,bbox _i -representing the item category and location box predicted by the i < th > prediction target; function H represents an anomaly classification module, P _i A set of probabilities representing that the ith item is predicted to be of each abnormal category;

3. a method of detecting an item in a container system according to any one of claims 1 and 2, the method comprising:

the detection model training device generates a detection model with set model parameters; the method specifically comprises the following steps:

the detection model training device carries out multi-target detection labeling on the training image to obtain a first labeling data set composed of labeling values of target goods; the labeling value comprises second image data of the target goods and second label information, wherein the second label information comprises category labels and position frame labels of the target goods;

the detection model training device adds the abnormal label of the kth abnormality in the label value of each abnormal target goods j in the first label data set to form an abnormal label value, and a second label data set comprising the abnormal labels is obtained;

The detection model training device sends the training image to the goods detection device, and generates prediction information and abnormal prediction information of the target goods i through a detection model with set model parameters;

the detection model training device calculates an abnormal classification loss function weighting coefficient; the method specifically comprises the following steps:

the detection model training device calculates the value of epsilon by using a formula (3); wherein n is the total number of all target goods in the second marked data set, m ₁ For the number of abnormal target goods in the set A, m ₂ The number of normal target goods in the set A;

the detection model training device calculates an abnormal classification loss function weighting coefficient by using the formula (2); wherein λ is an anomaly classification loss function weighting coefficient, ε is a fraction of approximately 0, x is an image in the dataset, and A is a set of images in the second annotation dataset with anomaly annotations;

the detection model training device calculates a joint loss function of the detection model according to the prediction information and the abnormal prediction information of the target goods generated by the goods detection device, the second labeling data set and the abnormal classification loss function weighting coefficient; the method specifically comprises the following steps:

Calculating a loss-of-good detection function loss _general ；

Calculating an abnormal classification loss function loss using formulas (4) to (8) _anomaly The method comprises the steps of carrying out a first treatment on the surface of the Wherein L (i) is used to calculate the loss of a single predicted target; iou (iou) _i Bbox representing the ith target item in the training image _i Maximum overlapping rate of the target goods position frames with the abnormal labels; y is _i,k An anomaly flag indicating whether the kth anomaly exists in the ith target article i, if y _i,k If y is equal to 1, the anomaly flag is present _i,k =0, then the anomaly note is not present; gtbox _j Position frame, bbox, of jth target goods with abnormal labels _i A location box representing a predicted ith target item; gtcls _j,k The kth anomaly label of the target goods which indicates the jth anomaly label exists or not, and id _i The target item number with the abnormal label, which has the largest overlapping rate with the ith target item i, is represented, j is the target item number with the abnormal label, j epsilon N ⁺ ；

Detecting a loss function loss from the item _general The abnormal classification loss function loss _anomaly And the anomaly classification loss function weighting coefficient λ, calculating the joint loss function loss using equation (9);

loss＝loss _general +λ·loss _anomaly (9)

when judging that the joint loss function is not converged, the detection model training device calculates the partial derivative of the current model parameter of the detection model based on a chain rule and the joint loss function, and subtracts the partial derivative from the current model parameter to obtain a new current model parameter of the detection model; ending training when the joint loss function is judged to be converged;

The goods detection device detects goods in the collected goods images of the goods in the container by utilizing the detection model with the set model parameters so as to generate detection results; the method specifically comprises the following steps:

the goods detection device carries out multi-target detection on the goods image by adopting a multi-target detection algorithm based on deep learning to obtain deep visual characteristics and prediction information of target goods; the prediction information comprises first image data of a target article and first tag information, wherein the first tag information comprises category prediction and position frame prediction of the target article;

the goods detection device uses a classification method to obtain an abnormal prediction probability set P of the target goods according to the depth visual characteristics _i ＝{p _i,k |k∈N ⁺ ,K∈N ⁺ And K is more than or equal to 1 and less than or equal to K; wherein K is the number of abnormal categories, K is the total number of abnormal categories, i is the number of target goods, i is N ⁺ ，N ⁺ Is a positive integer;

the goods detection device respectively judges the abnormal prediction probability set P _i Anomaly prediction probability p of K anomalies in a plurality _i,k Whether greater than a threshold;

the article detecting device predicts the abnormality probability p _i,k Adding tag information to the anomaly information of the kth anomaly greater than the threshold value to generate anomaly predictionInformation.

4. The article detecting method according to claim 3, wherein the article detecting device obtains the abnormal prediction probability set P of the target article from the depth visual feature using a classification method _i ＝{p _i,k |k∈N ⁺ ,K∈N ⁺ And K is more than or equal to 1 and less than or equal to K } ", specifically comprising:

the article detection device obtains an abnormal prediction probability set P of the target article according to the depth visual characteristics by using a formula (1) _i ＝{p _i,k |k∈N ⁺ ,K∈N ⁺ And K is more than or equal to 1 and less than or equal to K; where x is the input image and function F represents the candidate object extractor, h _i Features representing the ith predicted target; function G represents the item classification and prediction box component, { cls _i ,bbox _i -representing the item category and location box for which the i-th prediction target is predicted; function H represents an anomaly classification module, P _i A set of probabilities representing that the ith item is predicted to be of each abnormal category;

5. the goods detection device is used for collecting goods images of goods in a sales counter and is characterized by also being used for detecting the goods in the goods images by utilizing a detection model with preset model parameters so as to produce detection results; the detection model with the preset model parameters is obtained by training the detection model by a detection model training device; the article detecting device includes a first processing unit including:

the abnormality classification module is used for:

a parameter training module, comprising:

a loss function calculation sub-module for:

Calculating an anomaly classification loss function loss of the anomaly classification module by using formulas (4) to (8) _anomaly The method comprises the steps of carrying out a first treatment on the surface of the Wherein L (i) is used to calculate the loss of a single predicted target; iou (iou) _i Bbox representing the ith target item in the training image _i Maximum overlapping rate of the target goods position frames with the abnormal labels; y is _i,k An anomaly flag indicating whether the ith target article has the kth anomaly, if y _i,k If y is equal to 1, the anomaly flag is present _i,k =0, then the anomaly note is not present; wherein i is N ⁺ ，I∈N ⁺ I is not less than 1 and not more than I, I is the number of the target goods, I is the total number of the target goods, N ⁺ Is a positive integer; gtbox _j Position frame, bbox, of jth target goods with abnormal labels _i A location box representing a predicted ith target item; gtcls _j,k The kth anomaly label of the target goods which indicates the jth anomaly label exists or not, and id _i The target item number with the abnormal label, which has the largest overlapping rate with the ith target item i, is represented, j is the target item number with the abnormal label, j epsilon N ⁺ ；

loss＝loss _general +λ·loss _anomaly (9)

the model parameter setting submodule is used for calculating the partial derivative of the current model parameter of the detection model based on a chain rule and the joint loss function when the joint loss function is judged not to be converged by the loss function calculating submodule, and subtracting the partial derivative from the current model parameter to obtain a new current model parameter of the detection model; and the loss function calculation submodule is further used for ending training when the joint loss function is judged to be converged by the loss function calculation submodule;

The first processing unit is further configured to use a new current model parameter for actual goods detection when the model parameter setting submodule finishes training on the detection model.

6. The article detecting method applied to the article detecting apparatus according to claim 5, characterized in that the method comprises:

the goods detection device detects goods in the collected goods images of the goods in the container by using a detection model with set model parameters so as to generate a detection result; the method specifically comprises the following steps:

the goods detection device carries out multi-target detection on the goods image by using a multi-target detection algorithm based on deep learning to obtain deep visual characteristics and prediction information of target goods; the prediction information comprises first image data of a target article and first tag information, wherein the first tag information comprises category prediction and position frame prediction of the target article;

the article detecting device predicts the abnormality probability p _i,k Adding label information into the abnormal information of the kth abnormal which is larger than the threshold value to generate abnormal prediction information;

the detection model with set model parameters is generated by a detection model training device, and specifically comprises the following steps:

calculating a loss-of-good detection function loss _general ；

Calculating an abnormal classification loss function loss using formulas (4) to (8) _anomaly The method comprises the steps of carrying out a first treatment on the surface of the Wherein L (i) is used to calculate the loss of a single predicted target; iou (iou) _i Bbox representing the ith target item in the training image _i Maximum overlapping rate of the target goods position frames with the abnormal labels; y is _i,k An anomaly flag indicating whether the kth anomaly exists in the ith target article i, if y _i,k If y is equal to 1, the anomaly flag is present _i,k =0, thenIndicating that the anomaly note does not exist; gtbox _j Position frame, bbox, of jth target goods with abnormal labels _i A location box representing a predicted ith target item; gtcls _j,k The kth anomaly label of the target goods which indicates the jth anomaly label exists or not, and id _i The target item number with the abnormal label, which has the largest overlapping rate with the ith target item i, is represented, j is the target item number with the abnormal label, j epsilon N ⁺ ；

loss＝loss _general +λ·loss _anomaly (9)

when judging that the joint loss function is not converged, the detection model training device calculates the partial derivative of the current model parameter of the detection model based on a chain rule and the joint loss function, and subtracts the partial derivative from the current model parameter to obtain a new current model parameter of the detection model; and when the joint loss function is judged to be converged, ending training.