CN111597857B - Logistics package detection method, device, equipment and readable storage medium - Google Patents

Logistics package detection method, device, equipment and readable storage medium Download PDF

Info

Publication number
CN111597857B
CN111597857B CN201910126340.1A CN201910126340A CN111597857B CN 111597857 B CN111597857 B CN 111597857B CN 201910126340 A CN201910126340 A CN 201910126340A CN 111597857 B CN111597857 B CN 111597857B
Authority
CN
China
Prior art keywords
image
package
information
covering
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910126340.1A
Other languages
Chinese (zh)
Other versions
CN111597857A (en
Inventor
任子辉
许绍云
李功燕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhongke Weizhi Intelligent Manufacturing Technology Jiangsu Co ltd
Original Assignee
Zhongke Weizhi Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhongke Weizhi Technology Co ltd filed Critical Zhongke Weizhi Technology Co ltd
Priority to CN201910126340.1A priority Critical patent/CN111597857B/en
Publication of CN111597857A publication Critical patent/CN111597857A/en
Application granted granted Critical
Publication of CN111597857B publication Critical patent/CN111597857B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/10Terrestrial scenes
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B07SEPARATING SOLIDS FROM SOLIDS; SORTING
    • B07CPOSTAL SORTING; SORTING INDIVIDUAL ARTICLES, OR BULK MATERIAL FIT TO BE SORTED PIECE-MEAL, e.g. BY PICKING
    • B07C5/00Sorting according to a characteristic or feature of the articles or material being sorted, e.g. by control effected by devices which detect or measure such characteristic or feature; Sorting by manually actuated devices, e.g. switches
    • B07C5/34Sorting according to other particular properties
    • B07C5/342Sorting according to other particular properties according to optical properties, e.g. colour
    • B07C5/3422Sorting according to other particular properties according to optical properties, e.g. colour using video scanning devices, e.g. TV-cameras
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/045Combinations of networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/08Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
    • G06Q10/083Shipping
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/40Extraction of image or video features
    • G06V10/44Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00Indexing scheme relating to image or video recognition or understanding
    • G06V2201/06Recognition of objects for industrial automation

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Business, Economics & Management (AREA)
  • Economics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biomedical Technology (AREA)
  • Human Resources & Organizations (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Strategic Management (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Development Economics (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a logistics package detection method, which comprises the following steps: collecting an image of a parcel on a sorting conveyor; fusing a first data matrix formed by X coordinate values and a second data matrix formed by Y coordinate values of each coordinate in the image and gray values of the image into image information of the image; processing the image information through a preset detection model to obtain the category of the target contained in the image information and the position information of the boundary frame covering each target; the object included in the image information includes at least a package and an organ plate. The method can determine the position information of the boundary frame covering each target in the image, namely the position information of each target, and the coordinate information of the image is added in the detection process, so that the detection precision of the position of the logistics package on the sorting conveyor belt and the package sorting accuracy are improved. The logistics package detection device, the logistics package detection equipment and the readable storage medium have the same technical effects.

Description

Logistics package detection method, device, equipment and readable storage medium
Technical Field
The present invention relates to the field of image recognition technologies, and in particular, to a method, an apparatus, a device, and a readable storage medium for detecting a logistic package.
Background
In recent years, with the rapid development of electronic commerce and artificial intelligence technology, the logistics industry has come to be explosive. Therefore, the induction system is applied to an automation and intelligent system of a logistics scene. The automatic sorting system for the logistics packages is based on a sorting operation basic flow, integrates core technologies of the internet of things such as image recognition, automatic control, data communication and the like, and realizes standardized, standardized and intelligent sorting of the express packages. The express sorting efficiency and quality are effectively improved, and intelligent upgrading of the logistics supply chain is promoted.
Fig. 1 is a logistics package sorting system comprising: a package supply table, a package detection camera, a bar code identification camera, a plurality of sorting ports and a sorting conveyor belt; the sorting conveyor belt is formed by connecting trolleys and organ plates, and every two trolleys are connected through the organ plates. The workflow of the logistics package sorting system is as follows: the staff arranges the parcel, makes its face list place up, sends and supplies the package platform, when empty platform, will supply the parcel on the package platform to send the letter sorting conveyer belt, the face list promptly contains the commodity circulation information and discerns the commodity circulation list of barcode. When the package is conveyed to the position of the package detection camera by the sorting conveyor belt, the package detection camera detects the position of the package on the sorting conveyor belt so as to determine whether the position of the package is convenient for the bar code recognition camera to scan; when a package is transported by the sorting conveyor to the barcode recognition camera location, the barcode recognition camera scans the barcode of the package to determine to which sorting gate the current package is transported.
It can be seen that the main functions of the parcel detection camera are: the location of the parcel on the sorting conveyor is determined to determine if the parcel's location needs to be adjusted. In the existing parcel detection cameras, the position of the parcel is generally determined by adopting a traditional target detection algorithm, however, the traditional target detection algorithm is lack of the expression capability of the traditional target detection algorithm on image features (gray scale, color, texture and the like) because the traditional target detection algorithm depends on the image features designed manually; meanwhile, the traditional target detection algorithm only takes the gray value of the image as the image information of the image, and lacks the expression capability of the position information of the image, so that the detection precision and accuracy of the traditional target detection algorithm are reduced. Thus, if a conventional object detection algorithm is applied to the parcel detection camera, the accuracy of the position information of the parcel detected by the parcel detection camera will be insufficient, which may result in the parcel being delivered to the wrong sort mouth.
Therefore, how to improve the accuracy of detecting the position of the logistic package on the sorting conveyor belt is a problem that needs to be solved by those skilled in the art.
Disclosure of Invention
The invention aims to provide a method, a device and equipment for detecting logistics packages and a readable storage medium, so as to improve the position detection precision of the logistics packages on a sorting conveyor belt.
In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:
a method for detecting a logistic package, comprising:
collecting an image of a parcel on a sorting conveyor;
fusing a first data matrix formed by X coordinate values and a second data matrix formed by Y coordinate values of each coordinate in the image and gray values of the image into image information of the image;
processing the image information through a preset detection model to obtain the category of the target contained in the image information and the position information of a boundary box covering each target; wherein, the target included in the image information at least includes: package and organ plate.
The training step of the detection model comprises the following steps:
acquiring a training image, and extracting features of two scales of the training image through a convolution layer and a residual error network layer;
the extracted features are fused and analyzed through a feature processing layer, and an analysis result is obtained;
determining a detection result of the training image according to the analysis result, wherein the detection result is as follows: the detection category of the target contained in the training image, and the detection position information of the bounding box covering each target;
judging whether the difference between the detection result and the labeling information of the training image accords with a preset condition or not through a preset loss function, wherein the labeling information is the real category of each target contained in the training image and the real position information of a boundary frame covering each target;
if yes, the detection model training is completed;
the residual network layer is formed by arranging a plurality of network sublayers with different scales from large to small according to the scale, and the last two network sublayers in the residual network layer respectively output the characteristics of the training images with two scales.
The feature processing layer fuses and analyzes the extracted features to obtain an analysis result, and the method comprises the following steps:
convolving and upsampling a first feature output by a penultimate network sub-layer in the residual network layer to obtain a target feature with the same scale as a second feature output by a penultimate network sub-layer in the residual network layer;
connecting and convolving the target feature and the second feature to obtain a first vector;
convolving the first feature, and expressing a convolution result by vectors to obtain a second vector;
and analyzing the first vector and the second vector to obtain the analysis result.
Wherein, still include:
presetting the loss function according to the difference between the detection result and the labeling information, wherein the difference between the detection result and the labeling information at least comprises: the position error of the bounding box covering the same object, the confidence error of whether the object exists in the bounding box covering the same object, and the category error of the same object.
Wherein, after the training image is obtained, the method further comprises:
and recording the labeling information of the training image by using an xml file.
Wherein before the fusing of the gray value of the image into the image information of the image, the method further comprises:
and preprocessing the image, and converting the preprocessed image into a gray image.
The processing the image information through a preset detection model to obtain the category of the object contained in the image information and the position information of the bounding box covering each object, and then further include:
and determining whether the position of the package needs to be adjusted according to the position information of the bounding box covering the package in the image.
A logistic parcel detection apparatus comprising:
the acquisition module is used for acquiring images of the packages on the package sorting conveyor belt;
the fusion module is used for fusing the first data matrix formed by the X coordinate values and the second data matrix formed by the Y coordinate values of each coordinate in the image and the gray value of the image into the image information of the image;
the detection module is used for processing the image information through a preset detection model to obtain the category of the target contained in the image information and the position information of the boundary frame covering each target; wherein, the target included in the image information at least includes: package and organ plate.
A logistic parcel detection apparatus comprising:
a memory for storing a computer program;
and the processor is used for realizing the steps of the logistics package detection method when executing the computer program.
A readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the logistic parcel detection method of any one of the above.
According to the scheme, the logistics package detection method provided by the embodiment of the invention comprises the following steps: collecting an image of a parcel on a sorting conveyor; fusing a first data matrix formed by X coordinate values and a second data matrix formed by Y coordinate values of each coordinate in the image and gray values of the image into image information of the image; processing the image information through a preset detection model to obtain the category of the target contained in the image information and the position information of a boundary box covering each target; wherein, the target included in the image information at least includes: package and organ plate.
As can be seen, for the collected image of the package on the sorting conveyor, the method first fuses the first data matrix formed by the X-coordinate values and the second data matrix formed by the Y-coordinate values of each coordinate in the image, and the gray values of the image into the image information of the image, namely: the position information of the image is embedded in the image information in order to determine the position information of the object in the image. Further, processing the image information containing the coordinate information through a preset detection model to obtain the category of the target contained in the image information and the position information of the boundary box covering each target; wherein, the target contained in the image information at least comprises: package and organ plate. Therefore, the position information of the boundary frame of each target in the image, namely the position information of each target, is determined, namely the position information of the logistics package on the sorting conveyor belt is detected, and the detection accuracy of the position of the logistics package on the sorting conveyor belt is improved due to the fact that the coordinate information of the image is added in the detection process, so that the accuracy of package sorting can be improved.
Correspondingly, the logistics package detection device, the logistics package detection equipment and the readable storage medium have the technical effects.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic diagram of a logistics package sorting system according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method for detecting a logistic package according to an embodiment of the present invention;
FIG. 3 is a flowchart of a test model training method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a logistic parcel detecting apparatus according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a logistic package detection device according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a logistic package detection model according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a result of analysis of a package of a logistics package according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of image information of a logistic package according to an embodiment of the present invention;
FIG. 9 is a schematic diagram of error of position information of a bounding box covering an object in an image according to an embodiment of the present invention;
fig. 10 is a diagram showing a production test effect according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The embodiment of the invention discloses a method, a device and equipment for detecting logistics packages and a readable storage medium, which are used for improving the position detection precision of the logistics packages on a sorting conveyor belt.
Referring to fig. 2, the method for detecting a logistic package provided by the embodiment of the invention includes:
s201, acquiring an image of a package on a sorting conveyor belt;
s202, fusing a first data matrix formed by X coordinate values and a second data matrix formed by Y coordinate values of each coordinate in the image and gray values of the image into image information of the image;
wherein before fusing the first data matrix formed by the X coordinate values and the second data matrix formed by the Y coordinate values of each coordinate in the image and the gray values of the image into the image information of the image, the method further comprises: preprocessing an image, and converting the preprocessed image into a gray scale image.
S203, processing the image information through a preset detection model to obtain the category of the object contained in the image information and the position information of the boundary box covering each object.
Wherein, the target contained in the image information at least comprises: package and organ plate. The positional information of the bounding box includes: the length, width, and center point coordinates of the bounding box, or the coordinates of the four vertices of the bounding box.
Based on the application scene of the invention, the invention only needs to detect whether the position of the package is convenient for scanning by the bar code camera. When the package is positioned in the center of the trolley, the position is an ideal position, and the scanning angle is optimal; when the package is positioned off the center of the trolley, but still on the trolley, the package sorting system may control the trolley to roll so that the package is centered on the trolley; when the package is positioned at the edge of the organ plate or the sorting conveyor belt, the position of the package cannot be adjusted, so that the package falls into an incorrect sorting opening or the position of the package needs to be manually adjusted.
In the present embodiment, therefore, after determining the category of the object contained in the image and the positional information of the bounding box covering each object, the number and positions of the individual packages, the number and positions of the organ plates, and the like contained in the current image can be determined. Based on the obtained location information, it can be determined whether and how to adjust the location of the package.
The method comprises the steps of processing image information through a preset detection model, obtaining the category of the object contained in the image information and the position information of a boundary box covering each object, and then further comprising: and determining whether the position of the package needs to be adjusted according to the position information of the bounding box covering the package in the image.
Specifically, determining whether the position of the package needs to be adjusted according to the position information of the bounding box covering the package in the image includes: calculating the area of a boundary box covering each target in the image; and determines whether a portion of the parcel is outside the sorting conveyor.
When a part of the package is not positioned outside the sorting conveyor belt, judging whether an overlapping area exists between a boundary frame covering the current package and a boundary frame covering the organ plate for one package;
if yes, calculating an overlapping area M of the bounding box covering the current package and the bounding box covering the organ plate, and a sum N of the area of the bounding box covering the current package and the area of the bounding box covering the organ plate; and calculating the ratio of M to N-M, and when the ratio is larger than a preset threshold, considering that the position of the current package needs to be adjusted, generating corresponding adjustment information at the moment and sending the corresponding adjustment information to a management terminal for display so as to facilitate manual intervention of staff. Wherein N-M represents the difference between N and M.
Likewise, when a part of the package is positioned outside the sorting conveyor belt, judging whether an overlapping area exists between a boundary frame covering the current package and a boundary frame covering the outside of the sorting conveyor belt for one package;
if yes, calculating the overlapping area M of the boundary frame covering the current package and the boundary frame covering the outside of the sorting conveyor belt, and the sum N of the area of the boundary frame covering the current package and the area of the boundary frame covering the outside of the sorting conveyor belt; and calculating the ratio of M to N-M, and when the ratio is larger than a preset threshold value, considering that the position of the current package needs to be adjusted, and controlling the trolley of the sorting conveyor belt to roll at the moment so as to enable the current package to be positioned on the trolley. Wherein, the outside of letter sorting conveyer belt is the both sides position of letter sorting conveyer belt promptly.
It can be seen that this embodiment provides a logistic parcel detecting method, and the method first fuses, for an acquired image of a parcel on a sorting conveyor belt, a first data matrix formed by an X-coordinate value and a second data matrix formed by a Y-coordinate value of each coordinate in the image, and a gray value of the image into image information of the image, that is: the position information of the image is embedded in the image information in order to determine the position information of the object in the image. Further, processing the image information containing the coordinate information through a preset detection model to obtain the category of the target contained in the image information and the position information of the boundary box covering each target; wherein, the target contained in the image information at least comprises: package and organ plate. Therefore, the position information of the boundary frame of each target in the image, namely the position information of each target, is determined, namely the position information of the logistics package on the sorting conveyor belt is detected, and the detection accuracy of the position of the logistics package on the sorting conveyor belt is improved due to the fact that the coordinate information of the image is added in the detection process, so that the accuracy of package sorting can be improved.
Referring to fig. 3, the training steps of the detection model include:
s301, acquiring a training image, and extracting features of two scales of the training image through a convolution layer and a residual error network layer;
s302, fusing and analyzing the extracted features through a feature processing layer to obtain an analysis result;
s303, determining a detection result of the training image according to the analysis result;
wherein, the testing result is: training the detection category of the object contained in the image, and detecting position information of a boundary box covering each object;
s304, judging whether the difference between the detection result and the labeling information of the training image accords with a preset condition or not through a preset loss function; if yes, then execute S305; if not, executing S306;
the marking information is the real type of each target contained in the training image and the real position information of the boundary frame covering each target;
s305, detecting that model training is completed;
s306, updating parameters of the detection model through a random gradient descent method, and executing S301.
The residual network layer is formed by arranging a plurality of network sublayers with different scales from large to small according to the scale sizes, and the last two network sublayers in the residual network layer respectively output the characteristics of two scales of the training image.
The feature processing layer fuses and analyzes the extracted features to obtain an analysis result, and the method comprises the following steps:
convolving and upsampling a first feature output by a penultimate network sub-layer in the residual network layer to obtain a target feature with the same scale as a second feature output by the penultimate network sub-layer in the residual network layer;
connecting and convolving the target feature and the second feature to obtain a first vector;
convolving the first feature, and expressing a convolution result by vectors to obtain a second vector;
and analyzing the first vector and the second vector to obtain an analysis result.
Wherein, still include:
the loss function is preset according to the difference between the detection result and the labeling information, wherein the difference between the detection result and the labeling information at least comprises: the position error of the bounding box covering the same object, the confidence error of whether the object exists in the bounding box covering the same object, and the category error of the same object.
Wherein after the training image is acquired, the method further comprises: and recording the labeling information of the training image by using the xml file. Namely: the true class of each object in the image and the true position information of the bounding box covering each object are recorded.
The following describes a logistic package detection device provided by the embodiment of the present invention, and the logistic package detection device described below and the logistic package detection method described above can be referred to each other.
Referring to fig. 4, a logistic package detection device provided by an embodiment of the present invention includes:
an acquisition module 401 for acquiring images of packages located on the package sorting conveyor;
a fusion module 402, configured to fuse a first data matrix formed by an X coordinate value and a second data matrix formed by a Y coordinate value of each coordinate in the image, and a gray value of the image into image information of the image;
a detection module 403, configured to process the image information through a preset detection model, obtain a category of a target included in the image information, and position information of a bounding box covering each target; wherein, the target included in the image information at least includes: package and organ plate.
Wherein, still include: a training module for training the detection model, comprising:
the acquisition unit is used for acquiring a training image and extracting the characteristics of two scales of the training image through the convolution layer and the residual error network layer;
the analysis unit is used for fusing the extracted features through the feature processing layer and analyzing the features to obtain an analysis result;
the determining unit is used for determining a detection result of the training image according to the analysis result, wherein the detection result is as follows: the detection category of the target contained in the training image, and the detection position information of the bounding box covering each target;
the judging unit is used for judging whether the difference between the detection result and the labeling information of the training image accords with a preset condition or not through a preset loss function, wherein the labeling information is the real category of each target contained in the training image and the real position information of a boundary frame covering each target;
the completion unit is used for completing the training of the detection model when the difference between the detection result and the labeling information of the training image accords with a preset condition;
the residual network layer is formed by arranging a plurality of network sublayers with different scales from large to small according to the scale, and the last two network sublayers in the residual network layer respectively output the characteristics of the training images with two scales.
Wherein, the parsing unit includes:
the sampling subunit is used for convoluting and upsampling the first feature output by the penultimate network sublayer in the residual error network layer to obtain a target feature with the same scale as the second feature output by the penultimate network sublayer in the residual error network layer;
a first convolution subunit, configured to connect and convolve the target feature and the second feature to obtain a first vector;
a second convolution subunit, configured to convolve the first feature, and represent a convolution result with a vector to obtain a second vector;
and the analysis subunit is used for analyzing the first vector and the second vector to obtain the analysis result.
Wherein, training module still includes:
the presetting unit is used for presetting the loss function according to the difference between the detection result and the labeling information, wherein the difference between the detection result and the labeling information at least comprises: the position error of the bounding box covering the same object, the confidence error of whether the object exists in the bounding box covering the same object, and the category error of the same object.
Wherein, training module still includes:
and the recording unit is used for recording the labeling information of the training image by using the xml file.
Wherein, still include:
and the preprocessing unit is used for preprocessing the image and converting the preprocessed image into a gray image.
Wherein, still include:
and the determining module is used for determining whether the position of the package needs to be adjusted according to the position information of the bounding box covering the package in the image.
It can be seen that this embodiment provides a logistics parcel detecting device, including: the device comprises an acquisition module, a fusion module and a detection module. Firstly, an acquisition module acquires an image of a package on a package sorting conveyor belt; then a fusion module fuses a first data matrix formed by X coordinate values and a second data matrix formed by Y coordinate values of each coordinate in the image and gray values of the image into image information of the image; finally, the detection module processes the image information through a preset detection model to obtain the category of the target contained in the image information and the position information of the boundary frame covering each target; wherein, the target included in the image information at least includes: package and organ plate. Therefore, the modules work together, and each module performs the function, so that the detection precision of the position of the logistics package on the sorting conveyor belt and the accuracy of package sorting are improved.
The following describes a logistic package detection device provided by the embodiment of the present invention, and the logistic package detection device described below and the logistic package detection method and device described above can be referred to each other.
Referring to fig. 5, a logistics package detecting device provided by an embodiment of the present invention includes:
a memory 501 for storing a computer program;
a processor 502, configured to implement the steps of the method for detecting a logistic package according to any of the foregoing embodiments when executing the computer program.
The following describes a readable storage medium provided in the embodiments of the present invention, and the readable storage medium described below and the method, apparatus and device for detecting a logistic package described above may be referred to with each other.
A readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the logistic parcel detection method according to any of the embodiments described above.
The following test model can be designed based on the test model training step provided by the present invention, please refer to fig. 6. The detection model shown in fig. 6 includes: the device comprises a feature extraction module, a feature fusion module and an analysis module, wherein:
the feature extraction module comprises a convolution layer and five residual error network blocks, and feature scales extracted by the residual error network blocks are different;
the feature fusion module is used for convoluting the feature output by the last residual network block, then upsampling the feature to obtain a feature with a scale of 26x 256, connecting the feature with the feature output by the second last residual network block to obtain a feature with a scale of 26x 768, convoluting the feature with a scale of 26x 768 to obtain a feature with a scale of 13x 1024;
and the characteristic analysis module is used for respectively carrying out convolution on the two characteristics with the scale of 13x 1024 and analyzing the two obtained convolution results so as to output a data block.
The two obtained convolution results are analyzed, namely: the number and the category of the objects contained in the current image are determined, the position information of the bounding box covering each object, and the like.
The final output feature size of the feature extraction module is "13 x 1024", the last output size is "26 x 512", and these two features are input into the feature fusion module. The characteristic of 13x 1024 is divided into two paths, and one path is subjected to 3x3 convolution and 1x1 convolution to obtain 13x 21; the other path extends the data block into 26x 256 through the up-sampling layer, then merges with the second to last feature 26x 512 to obtain 26x 768, and connects the up-sampling result with the 26x 512, the features of "26 x 768" are obtained by convolving the features of "26 x 768" to obtain features with a scale of "13 x 1024", and the features of "13 x 1024" are convolved by 3x3 plus 1x1 to obtain "26 x 21". As such, two results are finally obtained: "26 x 21" and "13 x 21".
It should be noted that, since the small package and the large package have a relatively large difference in size and the organ plate has a size much larger than that of a general package, the feature extraction module has two-scale outputs, namely 13x13 scale and 26x26 scale. Wherein the 13x13 scale is used to detect large packages and organ plates and the 26x26 scale is used to detect small packages.
The two features of 26x 21 and 13x 21 pass through a feature analysis module, and the feature analysis module extracts category information, coordinate information and probability of existence of a target in a bounding box from a data block. For example: three bounding boxes, namely box1, box2 and box3, are predicted, and the information contained in each bounding box is as follows: x and y represent the abscissa and ordinate, respectively, of the central position of the bounding box, w and h represent the length and width, respectively, of the bounding box, p represents the probability that a target is present in the bounding box, and class1 and class2 represent the class of the target in that position. The analysis result is shown in fig. 7.
Before the image is input to the feature extraction module, a first data matrix formed by the X coordinate values and a second data matrix formed by the Y coordinate values of each coordinate in the image, and the gray values of the image are fused into image information of the image so as to embed position information in the image information. The image information is specifically shown in fig. 8.
In order to measure the error between the detection result and the true result contained in the output data block, namely: detecting the error between the position information of the boundary box covering the target A in the result and the position information of the boundary box covering the target A in the real result; detecting whether the confidence coefficient of the target exists in the boundary box covering the target A in the result and whether the confidence coefficient of the target exists in the boundary box covering the target A in the real result; and detecting errors between the determined class of the target A in the result and the class of the target A in the real result. The invention presets a loss function according to the difference of the three aspects, and specifically comprises the following steps:
1. the error of the positional information of the bounding box is represented by formula (1):
where S represents the number of meshes divided in the image, B represents the number of predicted bounding boxes,indicating whether the jth bounding box in the ith grid covers the current target, wherein the bounding box with the largest Iou of the true bounding box of the current target is a prediction bounding box covering the current target, please refer to fig. 9, wherein b=3, the predicted position is the position information of the bounding box covering the target a in the detection result, x 'and y' are the center point coordinates of the bounding box, w 'is the length of the bounding box, and h' is the width of the bounding box; the true position is the position information of the boundary box covering the target A in the labeling information, x and y are the coordinates of the central point of the boundary box, w is the length of the boundary box, and h is the width of the boundary box.
2. The confidence error is represented by equation (2):
wherein,,
3. the error of the category is expressed by the formula (3), and the category is two types: package and organ plate:
and (3) synthesizing 1, 2 and 3 to obtain a loss function which is shown as a formula (4):
Loss=Loss coord +Loss conf +Loss class (4)
in order to prove the superiority of the invention, the same batch of data is detected by adopting different detection methods, and the detection results are shown in table 1.
TABLE 1
As can be seen from Table 1, the accuracy of Faster R-CNN is higher, but the processing speed is slower; the accuracy of SSD, yolo v3 and tiny Yolo is slightly lower, but the processing speed is faster. The accuracy of the invention is closest to the fast R-CNN, and the processing speed is the fastest, so that the invention ensures the accuracy and improves the processing efficiency.
The invention has also been tested in an actual production environment. 28209 parcels are randomly selected from all the detection result graphs acquired within one month, and only 3 parcels are missed in the result graphs through manual inspection. That is, the false detection rate of the present invention is only one ten thousandth. A partial effect diagram of the production test is shown in fig. 10, with the pictures from three different sorting lines, respectively.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (9)

1. A method for detecting a logistic package, comprising:
collecting an image of a parcel on a sorting conveyor;
fusing a first data matrix formed by X coordinate values and a second data matrix formed by Y coordinate values of each coordinate in the image and gray values of the image into image information of the image;
processing the image information through a preset detection model to obtain the category of the target contained in the image information and the position information of a boundary box covering each target; wherein, the target included in the image information at least includes: the position information of the boundary box comprises the length, the width and the coordinates of a central point of the boundary box or the coordinates of four vertexes of the boundary box;
calculating the area of the bounding box covering each object in the image; judging whether a part of the package is positioned outside the sorting conveyor belt;
when a part of the package is not positioned outside the sorting conveyor belt, judging whether an overlapping area exists between a boundary frame covering the current package and a boundary frame covering the organ plate; if yes, calculating an overlapping area M of the bounding box covering the current package and the bounding box covering the organ plate, and a sum N of the area of the bounding box covering the current package and the area of the bounding box covering the organ plate; calculating the ratio of M to N-M; when the ratio is larger than a preset threshold value, determining that the position of the current package needs to be adjusted, generating corresponding adjustment information and sending the corresponding adjustment information to a management terminal for display;
when a part of the package is positioned outside the sorting conveyor belt, judging whether an overlapping area exists between a boundary frame covering the current package and the boundary frame covering the outside of the sorting conveyor belt; if yes, calculating the overlapping area M of the boundary frame covering the current package and the boundary frame covering the outside of the sorting conveyor belt, and the sum N of the area of the boundary frame covering the current package and the area of the boundary frame covering the outside of the sorting conveyor belt; and calculating the ratio of M to N-M, when the ratio is larger than a preset threshold value, determining that the position of the current package needs to be adjusted, and controlling the trolley of the sorting conveyor belt to roll so that the current package is positioned on the trolley.
2. The method of claim 1, wherein the training step of the detection model comprises:
acquiring a training image, and extracting features of two scales of the training image through a convolution layer and a residual error network layer;
the extracted features are fused and analyzed through a feature processing layer, and an analysis result is obtained;
determining a detection result of the training image according to the analysis result, wherein the detection result is as follows: the detection category of the target contained in the training image, and the detection position information of the bounding box covering each target;
judging whether the difference between the detection result and the labeling information of the training image accords with a preset condition or not through a preset loss function, wherein the labeling information is the real category of each target contained in the training image and the real position information of a boundary frame covering each target;
if yes, the detection model training is completed;
the residual network layer is formed by arranging a plurality of network sublayers with different scales from large to small according to the scale, and the last two network sublayers in the residual network layer respectively output the characteristics of the training images with two scales.
3. The method for detecting a logistic package according to claim 2, wherein the step of merging and analyzing the extracted features by the feature processing layer to obtain an analysis result includes:
convolving and upsampling a first feature output by a penultimate network sub-layer in the residual network layer to obtain a target feature with the same scale as a second feature output by a penultimate network sub-layer in the residual network layer;
connecting and convolving the target feature and the second feature to obtain a first vector;
convolving the first feature, and expressing a convolution result by vectors to obtain a second vector;
and analyzing the first vector and the second vector to obtain the analysis result.
4. The method of logistic parcel detection according to claim 3, further comprising:
presetting the loss function according to the difference between the detection result and the labeling information, wherein the difference between the detection result and the labeling information at least comprises: the position error of the bounding box covering the same object, the confidence error of whether the object exists in the bounding box covering the same object, and the category error of the same object.
5. The method for detecting a logistic parcel according to claim 4, further comprising, after the acquiring of the training image:
and recording the labeling information of the training image by using an xml file.
6. The method according to any one of claims 1 to 5, wherein before fusing the first data matrix formed by the X-coordinate values and the second data matrix formed by the Y-coordinate values of each coordinate in the image and the gray values of the image into the image information of the image, further comprising:
and preprocessing the image, and converting the preprocessed image into a gray image.
7. A logistic parcel detection device, comprising:
the acquisition module is used for acquiring images of the packages on the package sorting conveyor belt;
the fusion module is used for fusing the first data matrix formed by the X coordinate values and the second data matrix formed by the Y coordinate values of each coordinate in the image and the gray value of the image into the image information of the image;
the detection module is used for processing the image information through a preset detection model to obtain the category of the target contained in the image information and the position information of the boundary frame covering each target; wherein, the target included in the image information at least includes: the position information of the boundary box comprises the length, the width and the coordinates of a central point of the boundary box or the coordinates of four vertexes of the boundary box;
calculating the area of the bounding box covering each object in the image; judging whether a part of the package is positioned outside the sorting conveyor belt;
when a part of the package is not positioned outside the sorting conveyor belt, judging whether an overlapping area exists between a boundary frame covering the current package and a boundary frame covering the organ plate; if yes, calculating an overlapping area M of the bounding box covering the current package and the bounding box covering the organ plate, and a sum N of the area of the bounding box covering the current package and the area of the bounding box covering the organ plate; calculating the ratio of M to N-M; when the ratio is larger than a preset threshold value, determining that the position of the current package needs to be adjusted, generating corresponding adjustment information and sending the corresponding adjustment information to a management terminal for display;
when a part of the package is positioned outside the sorting conveyor belt, judging whether an overlapping area exists between a boundary frame covering the current package and the boundary frame covering the outside of the sorting conveyor belt; if yes, calculating the overlapping area M of the boundary frame covering the current package and the boundary frame covering the outside of the sorting conveyor belt, and the sum N of the area of the boundary frame covering the current package and the area of the boundary frame covering the outside of the sorting conveyor belt; and calculating the ratio of M to N-M, when the ratio is larger than a preset threshold value, determining that the position of the current package needs to be adjusted, and controlling the trolley of the sorting conveyor belt to roll so that the current package is positioned on the trolley.
8. A logistic parcel detection apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the method for detecting a logistic package as claimed in any one of claims 1 to 6 when executing said computer program.
9. A readable storage medium, wherein a computer program is stored on the readable storage medium, which when executed by a processor, implements the steps of the logistic parcel detection method according to any one of claims 1 to 6.
CN201910126340.1A 2019-02-20 2019-02-20 Logistics package detection method, device, equipment and readable storage medium Active CN111597857B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910126340.1A CN111597857B (en) 2019-02-20 2019-02-20 Logistics package detection method, device, equipment and readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910126340.1A CN111597857B (en) 2019-02-20 2019-02-20 Logistics package detection method, device, equipment and readable storage medium

Publications (2)

Publication Number Publication Date
CN111597857A CN111597857A (en) 2020-08-28
CN111597857B true CN111597857B (en) 2023-09-26

Family

ID=72188560

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910126340.1A Active CN111597857B (en) 2019-02-20 2019-02-20 Logistics package detection method, device, equipment and readable storage medium

Country Status (1)

Country Link
CN (1) CN111597857B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112934758B (en) * 2020-12-14 2022-10-14 中科院计算所西部高等技术研究院 Coal sorting hand-dialing control method based on image recognition
CN113554706B (en) * 2021-07-29 2024-02-27 中科微至科技股份有限公司 Trolley parcel position detection method based on deep learning
CN115035192B (en) * 2022-06-21 2023-04-14 北京远舢智能科技有限公司 Method and device for determining positions of tobacco leaf distributing vehicle and conveying belt
CN115496914B (en) * 2022-08-18 2023-05-02 成都智元汇信息技术股份有限公司 Method and device for extracting package of maximum outline based on column data

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108171748A (en) * 2018-01-23 2018-06-15 哈工大机器人(合肥)国际创新研究院 A kind of visual identity of object manipulator intelligent grabbing application and localization method
CN108876765A (en) * 2018-05-22 2018-11-23 塞伯睿机器人技术(长沙)有限公司 The target locating set and method of industrial sorting machine people
CN109344821A (en) * 2018-08-30 2019-02-15 西安电子科技大学 Small target detecting method based on Fusion Features and deep learning

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108171748A (en) * 2018-01-23 2018-06-15 哈工大机器人(合肥)国际创新研究院 A kind of visual identity of object manipulator intelligent grabbing application and localization method
CN108876765A (en) * 2018-05-22 2018-11-23 塞伯睿机器人技术(长沙)有限公司 The target locating set and method of industrial sorting machine people
CN109344821A (en) * 2018-08-30 2019-02-15 西安电子科技大学 Small target detecting method based on Fusion Features and deep learning

Also Published As

Publication number Publication date
CN111597857A (en) 2020-08-28

Similar Documents

Publication Publication Date Title
CN111597857B (en) Logistics package detection method, device, equipment and readable storage medium
CN107617573B (en) Logistics code identification and sorting method based on multitask deep learning
CN109583535B (en) Vision-based logistics barcode detection method and readable storage medium
CN110427793B (en) Bar code detection method and system based on deep learning
CN111461133B (en) Express delivery surface single item name identification method, device, equipment and storage medium
CN111062252B (en) Real-time dangerous goods semantic segmentation method, device and storage device
KR102283197B1 (en) A method and device for determining the type of product
CN111767902A (en) Method, device and equipment for identifying dangerous goods of security check machine and storage medium
CN113516146A (en) Data classification method, computer and readable storage medium
CN112819796A (en) Tobacco shred foreign matter identification method and equipment
CN109389105A (en) A kind of iris detection and viewpoint classification method based on multitask
CN111652541A (en) Industrial production monitoring method, system and computer readable storage medium
CN111428682A (en) Express sorting method, device, equipment and storage medium
KR102619659B1 (en) Cargo information recognition device and method to automate the Delivery cargo Handling
CN112052702A (en) Method and device for identifying two-dimensional code
CN117214178A (en) Intelligent identification method for appearance defects of package on packaging production line
CN111079575B (en) Material identification method and system based on package image characteristics
CN117132540A (en) PCB defect post-processing method based on segmentation model
US20230245293A1 (en) Failure detection and failure recovery for ai depalletizing
CN117853573A (en) Video processing method, device, electronic equipment and computer readable medium
JPWO2019003687A1 (en) Projection instructing device, luggage sorting system, and projection instructing method
CN112069841B (en) X-ray contraband parcel tracking method and device
CN111401104B (en) Classification model training method, classification method, device, equipment and storage medium
CN114596576A (en) Image processing method and device, electronic equipment and storage medium
CN113850167A (en) Commodity identification method and system based on edge calculation and machine deep learning

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information
CB02 Change of applicant information

Address after: 214105 No. 299 Dacheng Road, Xishan District, Jiangsu, Wuxi

Applicant after: Zhongke Weizhi intelligent manufacturing technology Jiangsu Co.,Ltd.

Address before: 214105 No. 299 Dacheng Road, Xishan District, Jiangsu, Wuxi

Applicant before: ZHONGKE WEIZHI INTELLIGENT MANUFACTURING TECHNOLOGY JIANGSU Co.,Ltd.

TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20200915

Address after: 214105 No. 299 Dacheng Road, Xishan District, Jiangsu, Wuxi

Applicant after: ZHONGKE WEIZHI INTELLIGENT MANUFACTURING TECHNOLOGY JIANGSU Co.,Ltd.

Address before: Zuchongzhi road Kunshan city 215347 Suzhou City, Jiangsu province No. 1699 building 7 floor

Applicant before: KUNSHAN BRANCH, INSTITUTE OF MICROELECTRONICS OF CHINESE ACADEMY OF SCIENCES

CB02 Change of applicant information
CB02 Change of applicant information

Address after: 214105 No. 979, Antai Third Road, Xishan District, Wuxi City, Jiangsu Province

Applicant after: Zhongke Weizhi Technology Co.,Ltd.

Address before: No. 299, Dacheng Road, Xishan District, Wuxi City, Jiangsu Province

Applicant before: Zhongke Weizhi intelligent manufacturing technology Jiangsu Co.,Ltd.

GR01 Patent grant
GR01 Patent grant