Disclosure of Invention
The invention aims to provide a method, a device and equipment for detecting logistics packages and a readable storage medium, so as to improve the position detection precision of the logistics packages on a sorting conveyor belt.
In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:
a method for detecting a logistic package, comprising:
collecting an image of a parcel on a sorting conveyor;
fusing a first data matrix formed by X coordinate values and a second data matrix formed by Y coordinate values of each coordinate in the image and gray values of the image into image information of the image;
processing the image information through a preset detection model to obtain the category of the target contained in the image information and the position information of a boundary box covering each target; wherein, the target included in the image information at least includes: package and organ plate.
The training step of the detection model comprises the following steps:
acquiring a training image, and extracting features of two scales of the training image through a convolution layer and a residual error network layer;
the extracted features are fused and analyzed through a feature processing layer, and an analysis result is obtained;
determining a detection result of the training image according to the analysis result, wherein the detection result is as follows: the detection category of the target contained in the training image, and the detection position information of the bounding box covering each target;
judging whether the difference between the detection result and the labeling information of the training image accords with a preset condition or not through a preset loss function, wherein the labeling information is the real category of each target contained in the training image and the real position information of a boundary frame covering each target;
if yes, the detection model training is completed;
the residual network layer is formed by arranging a plurality of network sublayers with different scales from large to small according to the scale, and the last two network sublayers in the residual network layer respectively output the characteristics of the training images with two scales.
The feature processing layer fuses and analyzes the extracted features to obtain an analysis result, and the method comprises the following steps:
convolving and upsampling a first feature output by a penultimate network sub-layer in the residual network layer to obtain a target feature with the same scale as a second feature output by a penultimate network sub-layer in the residual network layer;
connecting and convolving the target feature and the second feature to obtain a first vector;
convolving the first feature, and expressing a convolution result by vectors to obtain a second vector;
and analyzing the first vector and the second vector to obtain the analysis result.
Wherein, still include:
presetting the loss function according to the difference between the detection result and the labeling information, wherein the difference between the detection result and the labeling information at least comprises: the position error of the bounding box covering the same object, the confidence error of whether the object exists in the bounding box covering the same object, and the category error of the same object.
Wherein, after the training image is obtained, the method further comprises:
and recording the labeling information of the training image by using an xml file.
Wherein before the fusing of the gray value of the image into the image information of the image, the method further comprises:
and preprocessing the image, and converting the preprocessed image into a gray image.
The processing the image information through a preset detection model to obtain the category of the object contained in the image information and the position information of the bounding box covering each object, and then further include:
and determining whether the position of the package needs to be adjusted according to the position information of the bounding box covering the package in the image.
A logistic parcel detection apparatus comprising:
the acquisition module is used for acquiring images of the packages on the package sorting conveyor belt;
the fusion module is used for fusing the first data matrix formed by the X coordinate values and the second data matrix formed by the Y coordinate values of each coordinate in the image and the gray value of the image into the image information of the image;
the detection module is used for processing the image information through a preset detection model to obtain the category of the target contained in the image information and the position information of the boundary frame covering each target; wherein, the target included in the image information at least includes: package and organ plate.
A logistic parcel detection apparatus comprising:
a memory for storing a computer program;
and the processor is used for realizing the steps of the logistics package detection method when executing the computer program.
A readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the logistic parcel detection method of any one of the above.
According to the scheme, the logistics package detection method provided by the embodiment of the invention comprises the following steps: collecting an image of a parcel on a sorting conveyor; fusing a first data matrix formed by X coordinate values and a second data matrix formed by Y coordinate values of each coordinate in the image and gray values of the image into image information of the image; processing the image information through a preset detection model to obtain the category of the target contained in the image information and the position information of a boundary box covering each target; wherein, the target included in the image information at least includes: package and organ plate.
As can be seen, for the collected image of the package on the sorting conveyor, the method first fuses the first data matrix formed by the X-coordinate values and the second data matrix formed by the Y-coordinate values of each coordinate in the image, and the gray values of the image into the image information of the image, namely: the position information of the image is embedded in the image information in order to determine the position information of the object in the image. Further, processing the image information containing the coordinate information through a preset detection model to obtain the category of the target contained in the image information and the position information of the boundary box covering each target; wherein, the target contained in the image information at least comprises: package and organ plate. Therefore, the position information of the boundary frame of each target in the image, namely the position information of each target, is determined, namely the position information of the logistics package on the sorting conveyor belt is detected, and the detection accuracy of the position of the logistics package on the sorting conveyor belt is improved due to the fact that the coordinate information of the image is added in the detection process, so that the accuracy of package sorting can be improved.
Correspondingly, the logistics package detection device, the logistics package detection equipment and the readable storage medium have the technical effects.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The embodiment of the invention discloses a method, a device and equipment for detecting logistics packages and a readable storage medium, which are used for improving the position detection precision of the logistics packages on a sorting conveyor belt.
Referring to fig. 2, the method for detecting a logistic package provided by the embodiment of the invention includes:
s201, acquiring an image of a package on a sorting conveyor belt;
s202, fusing a first data matrix formed by X coordinate values and a second data matrix formed by Y coordinate values of each coordinate in the image and gray values of the image into image information of the image;
wherein before fusing the first data matrix formed by the X coordinate values and the second data matrix formed by the Y coordinate values of each coordinate in the image and the gray values of the image into the image information of the image, the method further comprises: preprocessing an image, and converting the preprocessed image into a gray scale image.
S203, processing the image information through a preset detection model to obtain the category of the object contained in the image information and the position information of the boundary box covering each object.
Wherein, the target contained in the image information at least comprises: package and organ plate. The positional information of the bounding box includes: the length, width, and center point coordinates of the bounding box, or the coordinates of the four vertices of the bounding box.
Based on the application scene of the invention, the invention only needs to detect whether the position of the package is convenient for scanning by the bar code camera. When the package is positioned in the center of the trolley, the position is an ideal position, and the scanning angle is optimal; when the package is positioned off the center of the trolley, but still on the trolley, the package sorting system may control the trolley to roll so that the package is centered on the trolley; when the package is positioned at the edge of the organ plate or the sorting conveyor belt, the position of the package cannot be adjusted, so that the package falls into an incorrect sorting opening or the position of the package needs to be manually adjusted.
In the present embodiment, therefore, after determining the category of the object contained in the image and the positional information of the bounding box covering each object, the number and positions of the individual packages, the number and positions of the organ plates, and the like contained in the current image can be determined. Based on the obtained location information, it can be determined whether and how to adjust the location of the package.
The method comprises the steps of processing image information through a preset detection model, obtaining the category of the object contained in the image information and the position information of a boundary box covering each object, and then further comprising: and determining whether the position of the package needs to be adjusted according to the position information of the bounding box covering the package in the image.
Specifically, determining whether the position of the package needs to be adjusted according to the position information of the bounding box covering the package in the image includes: calculating the area of a boundary box covering each target in the image; and determines whether a portion of the parcel is outside the sorting conveyor.
When a part of the package is not positioned outside the sorting conveyor belt, judging whether an overlapping area exists between a boundary frame covering the current package and a boundary frame covering the organ plate for one package;
if yes, calculating an overlapping area M of the bounding box covering the current package and the bounding box covering the organ plate, and a sum N of the area of the bounding box covering the current package and the area of the bounding box covering the organ plate; and calculating the ratio of M to N-M, and when the ratio is larger than a preset threshold, considering that the position of the current package needs to be adjusted, generating corresponding adjustment information at the moment and sending the corresponding adjustment information to a management terminal for display so as to facilitate manual intervention of staff. Wherein N-M represents the difference between N and M.
Likewise, when a part of the package is positioned outside the sorting conveyor belt, judging whether an overlapping area exists between a boundary frame covering the current package and a boundary frame covering the outside of the sorting conveyor belt for one package;
if yes, calculating the overlapping area M of the boundary frame covering the current package and the boundary frame covering the outside of the sorting conveyor belt, and the sum N of the area of the boundary frame covering the current package and the area of the boundary frame covering the outside of the sorting conveyor belt; and calculating the ratio of M to N-M, and when the ratio is larger than a preset threshold value, considering that the position of the current package needs to be adjusted, and controlling the trolley of the sorting conveyor belt to roll at the moment so as to enable the current package to be positioned on the trolley. Wherein, the outside of letter sorting conveyer belt is the both sides position of letter sorting conveyer belt promptly.
It can be seen that this embodiment provides a logistic parcel detecting method, and the method first fuses, for an acquired image of a parcel on a sorting conveyor belt, a first data matrix formed by an X-coordinate value and a second data matrix formed by a Y-coordinate value of each coordinate in the image, and a gray value of the image into image information of the image, that is: the position information of the image is embedded in the image information in order to determine the position information of the object in the image. Further, processing the image information containing the coordinate information through a preset detection model to obtain the category of the target contained in the image information and the position information of the boundary box covering each target; wherein, the target contained in the image information at least comprises: package and organ plate. Therefore, the position information of the boundary frame of each target in the image, namely the position information of each target, is determined, namely the position information of the logistics package on the sorting conveyor belt is detected, and the detection accuracy of the position of the logistics package on the sorting conveyor belt is improved due to the fact that the coordinate information of the image is added in the detection process, so that the accuracy of package sorting can be improved.
Referring to fig. 3, the training steps of the detection model include:
s301, acquiring a training image, and extracting features of two scales of the training image through a convolution layer and a residual error network layer;
s302, fusing and analyzing the extracted features through a feature processing layer to obtain an analysis result;
s303, determining a detection result of the training image according to the analysis result;
wherein, the testing result is: training the detection category of the object contained in the image, and detecting position information of a boundary box covering each object;
s304, judging whether the difference between the detection result and the labeling information of the training image accords with a preset condition or not through a preset loss function; if yes, then execute S305; if not, executing S306;
the marking information is the real type of each target contained in the training image and the real position information of the boundary frame covering each target;
s305, detecting that model training is completed;
s306, updating parameters of the detection model through a random gradient descent method, and executing S301.
The residual network layer is formed by arranging a plurality of network sublayers with different scales from large to small according to the scale sizes, and the last two network sublayers in the residual network layer respectively output the characteristics of two scales of the training image.
The feature processing layer fuses and analyzes the extracted features to obtain an analysis result, and the method comprises the following steps:
convolving and upsampling a first feature output by a penultimate network sub-layer in the residual network layer to obtain a target feature with the same scale as a second feature output by the penultimate network sub-layer in the residual network layer;
connecting and convolving the target feature and the second feature to obtain a first vector;
convolving the first feature, and expressing a convolution result by vectors to obtain a second vector;
and analyzing the first vector and the second vector to obtain an analysis result.
Wherein, still include:
the loss function is preset according to the difference between the detection result and the labeling information, wherein the difference between the detection result and the labeling information at least comprises: the position error of the bounding box covering the same object, the confidence error of whether the object exists in the bounding box covering the same object, and the category error of the same object.
Wherein after the training image is acquired, the method further comprises: and recording the labeling information of the training image by using the xml file. Namely: the true class of each object in the image and the true position information of the bounding box covering each object are recorded.
The following describes a logistic package detection device provided by the embodiment of the present invention, and the logistic package detection device described below and the logistic package detection method described above can be referred to each other.
Referring to fig. 4, a logistic package detection device provided by an embodiment of the present invention includes:
an acquisition module 401 for acquiring images of packages located on the package sorting conveyor;
a fusion module 402, configured to fuse a first data matrix formed by an X coordinate value and a second data matrix formed by a Y coordinate value of each coordinate in the image, and a gray value of the image into image information of the image;
a detection module 403, configured to process the image information through a preset detection model, obtain a category of a target included in the image information, and position information of a bounding box covering each target; wherein, the target included in the image information at least includes: package and organ plate.
Wherein, still include: a training module for training the detection model, comprising:
the acquisition unit is used for acquiring a training image and extracting the characteristics of two scales of the training image through the convolution layer and the residual error network layer;
the analysis unit is used for fusing the extracted features through the feature processing layer and analyzing the features to obtain an analysis result;
the determining unit is used for determining a detection result of the training image according to the analysis result, wherein the detection result is as follows: the detection category of the target contained in the training image, and the detection position information of the bounding box covering each target;
the judging unit is used for judging whether the difference between the detection result and the labeling information of the training image accords with a preset condition or not through a preset loss function, wherein the labeling information is the real category of each target contained in the training image and the real position information of a boundary frame covering each target;
the completion unit is used for completing the training of the detection model when the difference between the detection result and the labeling information of the training image accords with a preset condition;
the residual network layer is formed by arranging a plurality of network sublayers with different scales from large to small according to the scale, and the last two network sublayers in the residual network layer respectively output the characteristics of the training images with two scales.
Wherein, the parsing unit includes:
the sampling subunit is used for convoluting and upsampling the first feature output by the penultimate network sublayer in the residual error network layer to obtain a target feature with the same scale as the second feature output by the penultimate network sublayer in the residual error network layer;
a first convolution subunit, configured to connect and convolve the target feature and the second feature to obtain a first vector;
a second convolution subunit, configured to convolve the first feature, and represent a convolution result with a vector to obtain a second vector;
and the analysis subunit is used for analyzing the first vector and the second vector to obtain the analysis result.
Wherein, training module still includes:
the presetting unit is used for presetting the loss function according to the difference between the detection result and the labeling information, wherein the difference between the detection result and the labeling information at least comprises: the position error of the bounding box covering the same object, the confidence error of whether the object exists in the bounding box covering the same object, and the category error of the same object.
Wherein, training module still includes:
and the recording unit is used for recording the labeling information of the training image by using the xml file.
Wherein, still include:
and the preprocessing unit is used for preprocessing the image and converting the preprocessed image into a gray image.
Wherein, still include:
and the determining module is used for determining whether the position of the package needs to be adjusted according to the position information of the bounding box covering the package in the image.
It can be seen that this embodiment provides a logistics parcel detecting device, including: the device comprises an acquisition module, a fusion module and a detection module. Firstly, an acquisition module acquires an image of a package on a package sorting conveyor belt; then a fusion module fuses a first data matrix formed by X coordinate values and a second data matrix formed by Y coordinate values of each coordinate in the image and gray values of the image into image information of the image; finally, the detection module processes the image information through a preset detection model to obtain the category of the target contained in the image information and the position information of the boundary frame covering each target; wherein, the target included in the image information at least includes: package and organ plate. Therefore, the modules work together, and each module performs the function, so that the detection precision of the position of the logistics package on the sorting conveyor belt and the accuracy of package sorting are improved.
The following describes a logistic package detection device provided by the embodiment of the present invention, and the logistic package detection device described below and the logistic package detection method and device described above can be referred to each other.
Referring to fig. 5, a logistics package detecting device provided by an embodiment of the present invention includes:
a memory 501 for storing a computer program;
a processor 502, configured to implement the steps of the method for detecting a logistic package according to any of the foregoing embodiments when executing the computer program.
The following describes a readable storage medium provided in the embodiments of the present invention, and the readable storage medium described below and the method, apparatus and device for detecting a logistic package described above may be referred to with each other.
A readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the logistic parcel detection method according to any of the embodiments described above.
The following test model can be designed based on the test model training step provided by the present invention, please refer to fig. 6. The detection model shown in fig. 6 includes: the device comprises a feature extraction module, a feature fusion module and an analysis module, wherein:
the feature extraction module comprises a convolution layer and five residual error network blocks, and feature scales extracted by the residual error network blocks are different;
the feature fusion module is used for convoluting the feature output by the last residual network block, then upsampling the feature to obtain a feature with a scale of 26x 256, connecting the feature with the feature output by the second last residual network block to obtain a feature with a scale of 26x 768, convoluting the feature with a scale of 26x 768 to obtain a feature with a scale of 13x 1024;
and the characteristic analysis module is used for respectively carrying out convolution on the two characteristics with the scale of 13x 1024 and analyzing the two obtained convolution results so as to output a data block.
The two obtained convolution results are analyzed, namely: the number and the category of the objects contained in the current image are determined, the position information of the bounding box covering each object, and the like.
The final output feature size of the feature extraction module is "13 x 1024", the last output size is "26 x 512", and these two features are input into the feature fusion module. The characteristic of 13x 1024 is divided into two paths, and one path is subjected to 3x3 convolution and 1x1 convolution to obtain 13x 21; the other path extends the data block into 26x 256 through the up-sampling layer, then merges with the second to last feature 26x 512 to obtain 26x 768, and connects the up-sampling result with the 26x 512, the features of "26 x 768" are obtained by convolving the features of "26 x 768" to obtain features with a scale of "13 x 1024", and the features of "13 x 1024" are convolved by 3x3 plus 1x1 to obtain "26 x 21". As such, two results are finally obtained: "26 x 21" and "13 x 21".
It should be noted that, since the small package and the large package have a relatively large difference in size and the organ plate has a size much larger than that of a general package, the feature extraction module has two-scale outputs, namely 13x13 scale and 26x26 scale. Wherein the 13x13 scale is used to detect large packages and organ plates and the 26x26 scale is used to detect small packages.
The two features of 26x 21 and 13x 21 pass through a feature analysis module, and the feature analysis module extracts category information, coordinate information and probability of existence of a target in a bounding box from a data block. For example: three bounding boxes, namely box1, box2 and box3, are predicted, and the information contained in each bounding box is as follows: x and y represent the abscissa and ordinate, respectively, of the central position of the bounding box, w and h represent the length and width, respectively, of the bounding box, p represents the probability that a target is present in the bounding box, and class1 and class2 represent the class of the target in that position. The analysis result is shown in fig. 7.
Before the image is input to the feature extraction module, a first data matrix formed by the X coordinate values and a second data matrix formed by the Y coordinate values of each coordinate in the image, and the gray values of the image are fused into image information of the image so as to embed position information in the image information. The image information is specifically shown in fig. 8.
In order to measure the error between the detection result and the true result contained in the output data block, namely: detecting the error between the position information of the boundary box covering the target A in the result and the position information of the boundary box covering the target A in the real result; detecting whether the confidence coefficient of the target exists in the boundary box covering the target A in the result and whether the confidence coefficient of the target exists in the boundary box covering the target A in the real result; and detecting errors between the determined class of the target A in the result and the class of the target A in the real result. The invention presets a loss function according to the difference of the three aspects, and specifically comprises the following steps:
1. the error of the positional information of the bounding box is represented by formula (1):
where S represents the number of meshes divided in the image, B represents the number of predicted bounding boxes,indicating whether the jth bounding box in the ith grid covers the current target, wherein the bounding box with the largest Iou of the true bounding box of the current target is a prediction bounding box covering the current target, please refer to fig. 9, wherein b=3, the predicted position is the position information of the bounding box covering the target a in the detection result, x 'and y' are the center point coordinates of the bounding box, w 'is the length of the bounding box, and h' is the width of the bounding box; the true position is the position information of the boundary box covering the target A in the labeling information, x and y are the coordinates of the central point of the boundary box, w is the length of the boundary box, and h is the width of the boundary box.
2. The confidence error is represented by equation (2):
wherein,,
3. the error of the category is expressed by the formula (3), and the category is two types: package and organ plate:
and (3) synthesizing 1, 2 and 3 to obtain a loss function which is shown as a formula (4):
Loss=Loss coord +Loss conf +Loss class (4)
in order to prove the superiority of the invention, the same batch of data is detected by adopting different detection methods, and the detection results are shown in table 1.
TABLE 1
As can be seen from Table 1, the accuracy of Faster R-CNN is higher, but the processing speed is slower; the accuracy of SSD, yolo v3 and tiny Yolo is slightly lower, but the processing speed is faster. The accuracy of the invention is closest to the fast R-CNN, and the processing speed is the fastest, so that the invention ensures the accuracy and improves the processing efficiency.
The invention has also been tested in an actual production environment. 28209 parcels are randomly selected from all the detection result graphs acquired within one month, and only 3 parcels are missed in the result graphs through manual inspection. That is, the false detection rate of the present invention is only one ten thousandth. A partial effect diagram of the production test is shown in fig. 10, with the pictures from three different sorting lines, respectively.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.