CN111597857A - Logistics package detection method, device and equipment and readable storage medium - Google Patents

Logistics package detection method, device and equipment and readable storage medium Download PDF

Info

Publication number
CN111597857A
CN111597857A CN201910126340.1A CN201910126340A CN111597857A CN 111597857 A CN111597857 A CN 111597857A CN 201910126340 A CN201910126340 A CN 201910126340A CN 111597857 A CN111597857 A CN 111597857A
Authority
CN
China
Prior art keywords
image
information
detection
target
logistics
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910126340.1A
Other languages
Chinese (zh)
Other versions
CN111597857B (en
Inventor
任子辉
许绍云
李功燕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhongke Weizhi Intelligent Manufacturing Technology Jiangsu Co ltd
Original Assignee
Kunshan Branch Institute of Microelectronics of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Kunshan Branch Institute of Microelectronics of CAS filed Critical Kunshan Branch Institute of Microelectronics of CAS
Priority to CN201910126340.1A priority Critical patent/CN111597857B/en
Publication of CN111597857A publication Critical patent/CN111597857A/en
Application granted granted Critical
Publication of CN111597857B publication Critical patent/CN111597857B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B07—SEPARATING SOLIDS FROM SOLIDS; SORTING
    • B07C—POSTAL SORTING; SORTING INDIVIDUAL ARTICLES, OR BULK MATERIAL FIT TO BE SORTED PIECE-MEAL, e.g. BY PICKING
    • B07C5/00—Sorting according to a characteristic or feature of the articles or material being sorted, e.g. by control effected by devices which detect or measure such characteristic or feature; Sorting by manually actuated devices, e.g. switches
    • B07C5/34—Sorting according to other particular properties
    • B07C5/342—Sorting according to other particular properties according to optical properties, e.g. colour
    • B07C5/3422—Sorting according to other particular properties according to optical properties, e.g. colour using video scanning devices, e.g. TV-cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
    • G06Q10/083—Shipping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/06—Recognition of objects for industrial automation

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • Multimedia (AREA)
  • Economics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Development Economics (AREA)
  • Health & Medical Sciences (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Human Resources & Organizations (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Strategic Management (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a logistics package detection method, which comprises the following steps: collecting images of packages on a sorting conveyor belt; fusing a first data matrix formed by the X coordinate value and a second data matrix formed by the Y coordinate value of each coordinate in the image and the gray value of the image into image information of the image; processing the image information through a preset detection model to obtain the category of the targets contained in the image information and the position information of a boundary frame covering each target; the object included in the image information includes at least a parcel and a piano board. The method can determine the position information of the boundary frame covering each target in the image, namely the position information of each target, and the coordinate information of the image is added in the detection process, so that the detection precision of the position of the logistics parcel on the sorting conveyor belt and the accuracy of parcel sorting are improved. The logistics package detection device, the equipment and the readable storage medium disclosed by the invention also have the technical effects.

Description

Logistics package detection method, device and equipment and readable storage medium
Technical Field
The invention relates to the technical field of image recognition, in particular to a logistics package detection method, a logistics package detection device, logistics package detection equipment and a readable storage medium.
Background
In recent years, with the rapid development of electronic commerce and artificial intelligence technology, the logistics industry has been increased explosively. Therefore, the system is an automatic and intelligent system which is applied to logistics scenes in a mass mode. The automatic sorting system for the parcel packages of the express delivery integrates the core technologies of the internet of things such as image recognition, automatic control and data communication on the basis of the basic flow of sorting operation, and realizes the standardized, standardized and intelligent sorting of the parcel packages of the express delivery. Effectively promoted express delivery letter sorting efficiency and quality, promoted the intelligent upgrading of commodity circulation supply chain.
Fig. 1 is a logistics parcel sorting system comprising: the system comprises a bag supply table, a package detection camera, a bar code identification camera, a plurality of sorting ports and a sorting conveyor belt; the sorting conveyor belt is formed by connecting trolleys and a wind instrument board, and every two trolleys are connected through the wind instrument board. The work flow of the logistics package sorting system is as follows: the staff arranges the parcel well, makes its face list place up, send and supply a packet platform, when having empty platform truck, will supply the parcel on the packet platform to send the letter sorting conveyer belt, the face list contains the commodity circulation information promptly and discerns the commodity circulation list of bar code. When the packages are conveyed to the package detection camera position by the sorting conveyor belt, the package detection camera detects the position of the packages on the sorting conveyor belt so as to determine whether the position of the packages is convenient for the scanning of the bar code recognition camera; when a parcel is transported by the sorting conveyor to the barcode recognition camera location, the barcode recognition camera scans the barcode of the parcel to determine to which sorting gate the current parcel is transported.
It can be seen that the main roles of the package inspection camera are: the position of the package on the sorting conveyor is determined to determine if the package is in a position that requires adjustment. In the existing parcel detection camera, a traditional target detection algorithm is generally adopted to determine the position of a parcel, however, the traditional target detection algorithm depends on artificially designed image features (gray scale, color, texture and the like), so that the traditional target detection algorithm is deficient in the expression capability of the image features; meanwhile, the traditional target detection algorithm only takes the gray value of the image as the image information of the image, and lacks the expression capability of the position information of the image, so that the detection precision and accuracy of the traditional target detection algorithm are reduced. Thus, if a conventional object detection algorithm is applied to the package inspection camera, the accuracy of the position information of the package detected by the package inspection camera will be insufficient, which may result in the package being delivered to the wrong sorting gate.
Therefore, how to improve the position detection accuracy of the logistics packages on the sorting conveyor belt is a problem to be solved by those skilled in the art.
Disclosure of Invention
The invention aims to provide a method, a device and equipment for detecting logistics packages and a readable storage medium, which are used for improving the position detection precision of the logistics packages on a sorting conveyor belt.
In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:
a logistics package detection method comprises the following steps:
collecting images of packages on a sorting conveyor belt;
fusing a first data matrix formed by the X coordinate value and a second data matrix formed by the Y coordinate value of each coordinate in the image and the gray value of the image into image information of the image;
processing the image information through a preset detection model to obtain the types of the targets contained in the image information and the position information of a boundary frame covering each target; wherein the object contained in the image information at least comprises: a package and a piano plate.
Wherein the training step of the detection model comprises:
acquiring a training image, and extracting the characteristics of two scales of the training image through a convolution layer and a residual error network layer;
the extracted features are fused and analyzed through a feature processing layer to obtain an analysis result;
determining a detection result of the training image according to the analysis result, wherein the detection result is as follows: the training image comprises detection types of targets and detection position information of a boundary box covering each target;
judging whether the difference between the detection result and the labeling information of the training image meets a preset condition or not through a preset loss function, wherein the labeling information is the real category of each target contained in the training image and the real position information of a boundary box covering each target;
if so, finishing the training of the detection model;
the residual error network layer is formed by arranging a plurality of network sub-layers with different scales from large to small according to the scale size, and the last two network sub-layers in the residual error network layer respectively output the features of the training image with two scales.
The method for obtaining the analysis result by fusing and analyzing the extracted features through the feature processing layer comprises the following steps:
performing convolution and up-sampling on a first feature output by a first network sublayer in the residual error network layer to obtain a target feature with the same size as a second feature output by the second network sublayer in the residual error network layer;
connecting and convolving the target feature and the second feature to obtain a first vector;
performing convolution on the first characteristic, and representing a convolution result by using a vector to obtain a second vector;
and analyzing the first vector and the second vector to obtain the analysis result.
Wherein, still include:
presetting the loss function according to the difference between the detection result and the labeling information, wherein the difference between the detection result and the labeling information at least comprises: the position error of the bounding box covering the same target, the confidence error of whether the target exists in the bounding box covering the same target, and the category error of the same target.
After the training image is acquired, the method further includes:
and recording the marking information of the training image by using an xml file.
Before the fusing the first data matrix formed by the X coordinate values and the second data matrix formed by the Y coordinate values of each coordinate in the image and the gray level value of the image into the image information of the image, the method further comprises:
and preprocessing the image, and converting the preprocessed image into a gray image.
After the processing is performed on the image information through a preset detection model to obtain the categories of the targets included in the image information and the position information of the bounding box covering each target, the method further includes:
and determining whether the position of the parcel needs to be adjusted according to the position information of the bounding box covering the parcel in the image.
A logistics parcel detection apparatus comprising:
the acquisition module is used for acquiring the images of the packages on the package sorting conveyor belt;
the fusion module is used for fusing a first data matrix formed by X coordinate values of all coordinates in the image and a second data matrix formed by Y coordinate values, and the gray value of the image into image information of the image;
the detection module is used for processing the image information through a preset detection model to obtain the category of the targets contained in the image information and the position information of the boundary frame covering each target; wherein the object contained in the image information at least comprises: a package and a piano plate.
A logistics parcel detection apparatus comprising:
a memory for storing a computer program;
a processor, configured to implement the steps of the logistics package detection method according to any one of the above items when executing the computer program.
A readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps of the logistics package detection method of any one of the above.
According to the scheme, the logistics package detection method provided by the embodiment of the invention comprises the following steps: collecting images of packages on a sorting conveyor belt; fusing a first data matrix formed by the X coordinate value and a second data matrix formed by the Y coordinate value of each coordinate in the image and the gray value of the image into image information of the image; processing the image information through a preset detection model to obtain the category of the targets contained in the image information and the position information of a boundary frame covering each target; wherein the object contained in the image information at least comprises: a package and a piano plate.
As can be seen, for the acquired images of the packages on the sorting conveyor belt, the method first fuses a first data matrix formed by X coordinate values and a second data matrix formed by Y coordinate values of each coordinate in the images, and the gray values of the images into image information of the images, that is: the position information of the image is embedded in the image information to facilitate determination of the position information of the object in the image. Further, processing image information containing coordinate information through a preset detection model to obtain the category of the targets contained in the image information and position information of a boundary frame covering each target; wherein the object contained in the image information at least comprises: a package and a piano plate. Therefore, the position information of the boundary frame of each target in the image, namely the position information of each target can be determined, namely the position information of the logistics packages on the sorting conveyor belt is obtained through detection, and the coordinate information of the image is added in the detection process, so that the detection precision of the positions of the logistics packages on the sorting conveyor belt is improved, and the package sorting accuracy can be improved.
Correspondingly, the logistics package detection device, the equipment and the readable storage medium provided by the embodiment of the invention also have the technical effects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic diagram of a logistics parcel sorting system according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method for detecting logistics packages according to an embodiment of the present invention;
FIG. 3 is a flowchart of a method for training a test model according to an embodiment of the present invention;
FIG. 4 is a schematic view of a logistics parcel detection apparatus according to an embodiment of the present invention;
FIG. 5 is a schematic view of a logistics parcel detection apparatus according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a logistics package inspection model according to an embodiment of the present invention;
fig. 7 is a schematic diagram illustrating a logistics package parsing result according to an embodiment of the present invention;
FIG. 8 is a schematic view of image information of a logistics parcel according to an embodiment of the present invention;
FIG. 9 is a schematic diagram of an error in the position information of a bounding box covering an object in an image according to an embodiment of the present invention;
FIG. 10 is a diagram of the effect of a production test according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention discloses a method, a device and equipment for detecting logistics packages and a readable storage medium, which are used for improving the position detection precision of the logistics packages on a sorting conveyor belt.
Referring to fig. 2, a method for detecting a logistics package according to an embodiment of the present invention includes:
s201, collecting an image of a package on a sorting conveyor belt;
s202, fusing a first data matrix formed by X coordinate values of all coordinates in the image, a second data matrix formed by Y coordinate values and gray values of the image into image information of the image;
before fusing a first data matrix formed by an X coordinate value and a second data matrix formed by a Y coordinate value of each coordinate in the image and the gray value of the image into image information of the image, the method further comprises the following steps: and preprocessing the image, and converting the preprocessed image into a gray image.
S203, processing the image information through a preset detection model, and obtaining the category of the targets contained in the image information and the position information of the boundary frame covering each target.
Wherein the object contained in the image information at least comprises: a package and a piano plate. The position information of the bounding box includes: the length, width, and center point coordinates of the bounding box, or the coordinates of the four vertices of the bounding box.
Based on the application scene of the invention, the invention only needs to detect whether the position of the parcel is convenient for the scanning of the bar code camera. When the package is positioned in the center of the trolley, the position is an ideal position, and the scanning angle is optimal; when the position of the parcel deviates from the center of the trolley but is still on the trolley, the parcel sorting system can control the trolley to roll so that the parcel is positioned in the center of the trolley; and when the parcel is located the edge of wind instrument board or letter sorting conveyer belt, just can't adjust the position of parcel to can make the parcel fall into wrong letter sorting mouth, or need the position of manual adjustment parcel.
Therefore, in the present embodiment, after the object category included in the image and the position information of the bounding box covering each object are determined, the number and position of each parcel and the number and position of the piano plates included in the current image can be determined. From the obtained location information, it can be determined whether and how the location of the package needs to be adjusted.
The method includes the steps of processing image information through a preset detection model, obtaining the types of targets contained in the image information and position information of a boundary frame covering each target, and further including: and determining whether the position of the parcel needs to be adjusted according to the position information of the bounding box covering the parcel in the image.
Specifically, determining whether the position of the package needs to be adjusted according to the position information of the bounding box covering the package in the image includes: calculating the area of a bounding box covering each target in the image; and determines whether a portion of the package is outside of the sorting conveyor.
When a part of the packages is not positioned outside the sorting conveyor belt, judging whether an overlapping area exists between a boundary frame covering the current package and a boundary frame covering the piano board or not for one package;
if so, calculating the overlapping area M of the boundary frame covering the current parcel and the boundary frame covering the piano plate, and the sum N of the area of the boundary frame covering the current parcel and the area of the boundary frame covering the piano plate; and calculating the ratio of M to N-M, and when the ratio is greater than a preset threshold value, considering that the position of the current parcel needs to be adjusted, and at the moment, generating corresponding adjustment information and sending the adjustment information to a management terminal for displaying so that a worker can perform manual intervention. Wherein N-M represents the difference between N and M.
Similarly, when a part of the parcel is positioned outside the sorting conveyor belt, judging whether an overlapping area exists between a boundary frame covering the current parcel and a boundary frame covering the outside of the sorting conveyor belt or not for one parcel;
if so, calculating the overlapping area M of the boundary frame covering the current package and the boundary frame covering the outside of the sorting conveyor belt, and the sum N of the area of the boundary frame covering the current package and the area of the boundary frame covering the outside of the sorting conveyor belt; and calculating the ratio of M to N-M, and when the ratio is greater than a preset threshold value, determining that the position of the current parcel needs to be adjusted, and controlling the trolley of the sorting conveyor belt to roll so as to enable the current parcel to be positioned on the trolley. Wherein, the outside of the sorting conveyor belt is the position of both sides of the sorting conveyor belt.
As can be seen, the present embodiment provides a method for detecting a logistics package, in which for an acquired image of a package on a sorting conveyor belt, a first data matrix formed by an X coordinate value and a second data matrix formed by a Y coordinate value of each coordinate in the image, and a gray value of the image are fused into image information of the image, that is: the position information of the image is embedded in the image information to facilitate determination of the position information of the object in the image. Further, processing image information containing coordinate information through a preset detection model to obtain the category of the targets contained in the image information and position information of a boundary frame covering each target; wherein the object contained in the image information at least comprises: a package and a piano plate. Therefore, the position information of the boundary frame of each target in the image, namely the position information of each target can be determined, namely the position information of the logistics packages on the sorting conveyor belt is obtained through detection, and the coordinate information of the image is added in the detection process, so that the detection precision of the positions of the logistics packages on the sorting conveyor belt is improved, and the package sorting accuracy can be improved.
Referring to fig. 3, the training step of the detection model includes:
s301, acquiring a training image, and extracting features of two scales of the training image through a convolutional layer and a residual network layer;
s302, fusing and analyzing the extracted features through a feature processing layer to obtain an analysis result;
s303, determining a detection result of the training image according to the analysis result;
wherein, the detection result is as follows: training the detection category of the target contained in the image, and the detection position information of the boundary box covering each target;
s304, judging whether the difference between the detection result and the labeling information of the training image meets a preset condition or not through a preset loss function; if yes, go to S305; if not, executing S306;
the labeling information is the real category of each target contained in the training image and the real position information of a boundary box covering each target;
s305, completing the training of the detection model;
s306, updating the parameters of the detection model by a random gradient descent method, and executing S301.
The residual error network layer is composed of a plurality of network sublayers with different scales from large to small according to the scale size, and the last two network sublayers in the residual error network layer respectively output the features of two scales of the training image.
The extracted features are fused and analyzed through the feature processing layer, and an analysis result is obtained, wherein the method comprises the following steps:
performing convolution and sampling on a first feature output by a first network sublayer in the residual error network layer to obtain a target feature with the same scale as a second feature output by the second network sublayer in the residual error network layer;
connecting and convolving the target feature and the second feature to obtain a first vector;
performing convolution on the first characteristic, and expressing a convolution result by using a vector to obtain a second vector;
and analyzing the first vector and the second vector to obtain an analysis result.
Wherein, still include:
presetting a loss function according to the difference between the detection result and the labeling information, wherein the difference between the detection result and the labeling information at least comprises the following steps: the position error of the bounding box covering the same target, the confidence error of whether the target exists in the bounding box covering the same target, and the category error of the same target.
Wherein, after obtaining the training image, still include: and recording the annotation information of the training image by using the xml file. Namely: the true category of each object in the image and the true position information of the bounding box covering each object are recorded.
In the following, a logistics package detection device provided by an embodiment of the present invention is introduced, and a logistics package detection device described below and a logistics package detection method described above may be referred to each other.
Referring to fig. 4, a logistics package detection apparatus provided in an embodiment of the present invention includes:
an acquisition module 401, configured to acquire an image of a package on a package sorting conveyor;
a fusion module 402, configured to fuse a first data matrix formed by an X coordinate value and a second data matrix formed by a Y coordinate value of each coordinate in the image, and a gray value of the image into image information of the image;
a detection module 403, configured to process the image information through a preset detection model, to obtain a category of an object included in the image information and position information of a bounding box covering each object; wherein the object contained in the image information at least comprises: a package and a piano plate.
Wherein, still include: a training module for training the detection model, comprising:
the acquisition unit is used for acquiring a training image and extracting the characteristics of two scales of the training image through a convolutional layer and a residual network layer;
the analysis unit is used for fusing and analyzing the extracted features through the feature processing layer to obtain an analysis result;
a determining unit, configured to determine a detection result of the training image according to the analysis result, where the detection result is: the detection type of the targets contained in the training image and the detection position information of the boundary box covering each target;
a judging unit, configured to judge, through a preset loss function, whether a difference between the detection result and annotation information of the training image meets a preset condition, where the annotation information is a real category of each target included in the training image and real position information of a bounding box covering each target;
the completion unit is used for completing the training of the detection model when the difference between the detection result and the labeling information of the training image meets the preset condition;
the residual error network layer is formed by arranging a plurality of network sub-layers with different scales from large to small according to the scale size, and the last two network sub-layers in the residual error network layer respectively output the features of the training image with two scales.
Wherein the parsing unit includes:
the sampling subunit is used for performing convolution and up-sampling on a first feature output by a first network sublayer to the last in the residual error network layer to obtain a target feature with the same scale as a second feature output by the second network sublayer to the last in the residual error network layer;
the first convolution subunit is used for connecting and convolving the target feature and the second feature to obtain a first vector;
the second convolution subunit is used for performing convolution on the first feature and expressing a convolution result by using a vector to obtain a second vector;
and the analysis subunit is used for analyzing the first vector and the second vector to obtain the analysis result.
Wherein the training module further comprises:
a presetting unit, configured to preset the loss function according to a difference between the detection result and the labeled information, where the difference between the detection result and the labeled information at least includes: the position error of the bounding box covering the same target, the confidence error of whether the target exists in the bounding box covering the same target, and the category error of the same target.
Wherein the training module further comprises:
and the recording unit is used for recording the marking information of the training image by using the xml file.
Wherein, still include:
and the preprocessing unit is used for preprocessing the image and converting the preprocessed image into a gray image.
Wherein, still include:
and the determining module is used for determining whether the position of the parcel needs to be adjusted according to the position information of the bounding box covering the parcel in the image.
It can be seen that this embodiment provides a commodity circulation parcel detection device, includes: the device comprises an acquisition module, a fusion module and a detection module. Firstly, an acquisition module acquires an image of a package on a package sorting conveyor belt; then a fusion module fuses a first data matrix formed by the X coordinate value and a second data matrix formed by the Y coordinate value of each coordinate in the image and the gray value of the image into image information of the image; finally, the detection module processes the image information through a preset detection model to obtain the category of the targets contained in the image information and the position information of the boundary frame covering each target; wherein the object contained in the image information at least comprises: a package and a piano plate. Therefore, all the modules work in a cooperative manner, and each module plays its own role, so that the detection precision of the position of the logistics packages on the sorting conveyor belt and the accuracy of package sorting are improved.
In the following, a logistics package detection device provided by an embodiment of the present invention is introduced, and a logistics package detection device described below and a logistics package detection method and device described above may be referred to each other.
Referring to fig. 5, a logistics package detection apparatus provided in an embodiment of the present invention includes:
a memory 501 for storing a computer program;
a processor 502, configured to implement the steps of the logistics package detection method according to any of the above embodiments when executing the computer program.
The following describes a readable storage medium provided by an embodiment of the present invention, and a readable storage medium described below and a logistics package detection method, apparatus, and device described above may be referred to each other.
A readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of the logistics package detection method according to any of the above embodiments.
The following detection model can be designed based on the detection model training procedure provided by the present invention, please refer to fig. 6. The detection model shown in fig. 6 includes: the device comprises a feature extraction module, a feature fusion module and an analysis module, wherein:
the characteristic extraction module comprises a convolution layer and five residual error network blocks, and the characteristic scales extracted by the residual error network blocks are different;
the feature fusion module is used for performing convolution on the features output by the last residual network block, sampling the features to obtain the features with the scale of 26 × 256, connecting the features with the features output by the last residual network block to obtain the features with the scale of 26 × 768, and performing convolution on the features with the scale of 26 × 768 to obtain the features with the scale of 13 × 1024;
and the feature analysis module is used for respectively convolving the two features with the scales of 13 × 1024 and analyzing the two obtained convolution results so as to output the data block.
Wherein, the two obtained convolution results are analyzed, namely: the number and category of targets contained in the current image, the position information of the bounding box covering each target, and the like are determined.
The feature extraction module outputs the feature size of 13 × 1024 "at the end, and the previous layer outputs the feature size of 26 × 512", and the two features are input into the feature fusion module. Wherein, the characteristic of '13 × 1024' is divided into two paths, and one path is convoluted by 3x3 and 1x1 to obtain '13 × 21'; and the other path expands the data block into 26 × 256 through the upsampling layer, then fuses with the second reciprocal feature 26 × 512 to obtain 26 × 768 ", connects the upsampling result with 26 × 512 to obtain 26 × 768", and performs convolution on the feature of 26 × 768 to obtain a feature with the scale of 13 × 1024 ", and performs convolution on the feature of 13 × 1024" through 3 × 3 and 1 × 1 to obtain 26 × 21 ". In this way, two results are obtained: "26 × 21" and "13 × 21".
It should be noted that, because the sizes of the small package and the large package are different from each other, and the size of the gusset is much larger than that of the general package, there are two scales of outputs of the feature extraction module, namely, the 13x13 scale and the 26x26 scale. Wherein the 13x13 scale is used to detect large packages and piano plates, and the 26x26 scale is used to detect small packages.
The two features, 26 × 21 and 13 × 21, pass through a feature analysis module, which extracts category information, coordinate information, and a probability of the presence of a target in the bounding box from the data block. For example: three bounding boxes are obtained through prediction, namely box1, box2 and box3, and the information contained in each bounding box is as follows: x and y represent the abscissa and ordinate, respectively, of the center position of the bounding box, w and h represent the length and width, respectively, of the bounding box, p represents the probability that an object is present within the bounding box, and class1 and class2 represent the class of the object at that location. See FIG. 7 for the analysis results.
Before the image is input into the feature extraction module, a first data matrix composed of X coordinate values and a second data matrix composed of Y coordinate values of each coordinate in the image, and a gray scale value of the image are fused into image information of the image, so as to embed position information in the image information. The image information is shown in fig. 8.
In order to measure the error between the detection result and the real result contained in the output data block, namely: the error between the position information of the boundary frame covering the target A in the detection result and the position information of the boundary frame covering the target A in the real result; the confidence degree of whether the target exists in the boundary box covering the target A in the detection result and the error of the confidence degree of whether the target exists in the boundary box covering the target A in the real result are detected; the class of the object a determined in the detection result and the class of the object a in the real result. The invention presets a loss function according to the difference of the three aspects, and specifically comprises the following steps:
1. the error of the position information of the bounding box is represented by formula (1):
Figure BDA0001973719580000121
where S denotes the number of meshes divided in the image, B denotes the number of predicted bounding boxes,
Figure BDA0001973719580000122
indicating whether the jth bounding box in the ith grid covers the current target, wherein a bounding box with a maximum value of Iou from the real bounding box of the current target is a predicted bounding box covering the current target, please refer to fig. 9, where B is 3, the predicted position is the position information of the bounding box covering the target a in the detection result, x 'and y' are coordinates of a center point of the bounding box, w 'is the length of the bounding box, and h' is the width of the bounding box; the real position is the position information of the boundary box covering the target A in the marking information, x and y are the coordinates of the central point of the boundary box, w is the length of the boundary box, and h is the width of the boundary box.
2. The error of the confidence is represented by equation (2):
Figure BDA0001973719580000131
wherein,
Figure BDA0001973719580000132
3. the error of the category is expressed by formula (3), and the category is two types: wrapping and piano plate:
Figure BDA0001973719580000133
the obtained loss function is the formula (4) by integrating 1, 2 and 3:
Loss＝Losscoord+Lossconf+Lossclass(4)
in order to prove the superiority of the invention, different detection methods are adopted to detect the same batch of data, and the detection results are shown in table 1.
TABLE 1
Figure BDA0001973719580000134
As can be seen from Table 1, Faster R-CNN has higher accuracy but slower processing speed; SSD, Yolo v3, and tiny Yolo have slightly lower accuracy, but faster processing speeds. The accuracy of the invention is closest to that of fast R-CNN, and the processing speed is fastest, thus the invention can ensure the accuracy and improve the processing efficiency.
The present invention was also tested in a practical production environment. 28209 were randomly selected from all the test result graphs collected within one month, and only 3 of the result graphs were missed by manual inspection. That is, the false detection rate of the present invention is only one ten thousandth. The partial effect diagram of the production test is shown in fig. 10, and the pictures are respectively from three different sorting lines.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A logistics package detection method is characterized by comprising the following steps:
collecting images of packages on a sorting conveyor belt;
fusing a first data matrix formed by the X coordinate value and a second data matrix formed by the Y coordinate value of each coordinate in the image and the gray value of the image into image information of the image;
processing the image information through a preset detection model to obtain the category of the targets contained in the image information and the position information of a boundary frame covering each target; wherein the object contained in the image information at least comprises: a package and a piano plate.
2. The method of claim 1, wherein the training step of the inspection model comprises:
acquiring a training image, and extracting the characteristics of two scales of the training image through a convolution layer and a residual error network layer;
the extracted features are fused and analyzed through a feature processing layer to obtain an analysis result;
determining a detection result of the training image according to the analysis result, wherein the detection result is as follows: the detection type of the targets contained in the training image and the detection position information of the boundary box covering each target;
judging whether the difference between the detection result and the labeling information of the training image meets a preset condition or not through a preset loss function, wherein the labeling information is the real category of each target contained in the training image and the real position information of a boundary frame covering each target;
if so, finishing the training of the detection model;
the residual error network layer is formed by arranging a plurality of network sub-layers with different scales from large to small according to the scale size, and the last two network sub-layers in the residual error network layer respectively output the features of the training image with two scales.
3. The logistics package detection method of claim 2, wherein the fusing and analyzing the extracted features through the feature processing layer to obtain an analysis result comprises:
convolving and upsampling a first feature output by a penultimate network sublayer in the residual error network layer to obtain a target feature with the same scale as a second feature output by the penultimate network sublayer in the residual error network layer;
connecting and convolving the target feature and the second feature to obtain a first vector;
performing convolution on the first characteristic, and representing a convolution result by using a vector to obtain a second vector;
and analyzing the first vector and the second vector to obtain the analysis result.
4. The method of detecting logistics packages of claim 3, further comprising:
presetting the loss function according to the difference between the detection result and the labeling information, wherein the difference between the detection result and the labeling information at least comprises: the position error of the bounding box covering the same target, the confidence error of whether the target exists in the bounding box covering the same target, and the category error of the same target.
5. The method for detecting logistics packages of claim 4, wherein after the obtaining of the training image, further comprising:
and recording the marking information of the training image by using an xml file.
6. The logistics package detection method of any one of claims 1 to 5, wherein before integrating the first data matrix formed by X-coordinate values and the second data matrix formed by Y-coordinate values of each coordinate in the image and the gray-level values of the image into the image information of the image, the method further comprises:
and preprocessing the image, and converting the preprocessed image into a gray image.
7. The method for detecting logistics packages according to any one of claims 1 to 5, wherein after the processing the image information through a preset detection model to obtain the category of the objects contained in the image information and the position information of the bounding box covering each object, the method further comprises:
and determining whether the position of the parcel needs to be adjusted according to the position information of the bounding box covering the parcel in the image.
8. A logistics package detection device, comprising:
the acquisition module is used for acquiring the images of the packages on the package sorting conveyor belt;
the fusion module is used for fusing a first data matrix formed by X coordinate values and a second data matrix formed by Y coordinate values of each coordinate in the image and the gray value of the image into image information of the image;
the detection module is used for processing the image information through a preset detection model to obtain the category of the targets contained in the image information and the position information of the boundary frame covering each target; wherein the object contained in the image information at least comprises: a package and a piano plate.
9. A logistics package detection device, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the logistics parcel detection method of any one of claims 1-7 when executing said computer program.
10. A readable storage medium, characterized in that the readable storage medium has stored thereon a computer program which, when being executed by a processor, realizes the steps of the logistics package detection method according to any one of claims 1-7.
CN201910126340.1A 2019-02-20 2019-02-20 Logistics package detection method, device, equipment and readable storage medium Active CN111597857B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910126340.1A CN111597857B (en) 2019-02-20 2019-02-20 Logistics package detection method, device, equipment and readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910126340.1A CN111597857B (en) 2019-02-20 2019-02-20 Logistics package detection method, device, equipment and readable storage medium

Publications (2)

Publication Number Publication Date
CN111597857A true CN111597857A (en) 2020-08-28
CN111597857B CN111597857B (en) 2023-09-26

Family

ID=72188560

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910126340.1A Active CN111597857B (en) 2019-02-20 2019-02-20 Logistics package detection method, device, equipment and readable storage medium

Country Status (1)

Country Link
CN (1) CN111597857B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112934758A (en) * 2020-12-14 2021-06-11 中科院计算所西部高等技术研究院 Coal sorting hand-dialing control method based on image recognition
CN113554706A (en) * 2021-07-29 2021-10-26 中科微至智能制造科技江苏股份有限公司 Trolley package position detection method based on deep learning
CN115035192A (en) * 2022-06-21 2022-09-09 北京远舢智能科技有限公司 Method and device for determining positions of tobacco leaf distributing vehicle and conveying belt
CN115496914A (en) * 2022-08-18 2022-12-20 成都智元汇信息技术股份有限公司 Maximum outline package extraction method and device based on column data

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108171748A (en) * 2018-01-23 2018-06-15 哈工大机器人（合肥）国际创新研究院 A kind of visual identity of object manipulator intelligent grabbing application and localization method
CN108876765A (en) * 2018-05-22 2018-11-23 塞伯睿机器人技术（长沙）有限公司 The target locating set and method of industrial sorting machine people
CN109344821A (en) * 2018-08-30 2019-02-15 西安电子科技大学 Small target detecting method based on Fusion Features and deep learning

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108171748A (en) * 2018-01-23 2018-06-15 哈工大机器人（合肥）国际创新研究院 A kind of visual identity of object manipulator intelligent grabbing application and localization method
CN108876765A (en) * 2018-05-22 2018-11-23 塞伯睿机器人技术（长沙）有限公司 The target locating set and method of industrial sorting machine people
CN109344821A (en) * 2018-08-30 2019-02-15 西安电子科技大学 Small target detecting method based on Fusion Features and deep learning

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112934758A (en) * 2020-12-14 2021-06-11 中科院计算所西部高等技术研究院 Coal sorting hand-dialing control method based on image recognition
CN113554706A (en) * 2021-07-29 2021-10-26 中科微至智能制造科技江苏股份有限公司 Trolley package position detection method based on deep learning
CN113554706B (en) * 2021-07-29 2024-02-27 中科微至科技股份有限公司 Trolley parcel position detection method based on deep learning
CN115035192A (en) * 2022-06-21 2022-09-09 北京远舢智能科技有限公司 Method and device for determining positions of tobacco leaf distributing vehicle and conveying belt
CN115035192B (en) * 2022-06-21 2023-04-14 北京远舢智能科技有限公司 Method and device for determining positions of tobacco leaf distributing vehicle and conveying belt
CN115496914A (en) * 2022-08-18 2022-12-20 成都智元汇信息技术股份有限公司 Maximum outline package extraction method and device based on column data
CN115496914B (en) * 2022-08-18 2023-05-02 成都智元汇信息技术股份有限公司 Method and device for extracting package of maximum outline based on column data

Also Published As

Publication number Publication date
CN111597857B (en) 2023-09-26

Similar Documents

Publication Publication Date Title
CN111597857B (en) Logistics package detection method, device, equipment and readable storage medium
CN107617573B (en) Logistics code identification and sorting method based on multitask deep learning
CN113139559B (en) Training method of target detection model, and data labeling method and device
CN109092696B (en) Sorting system and sorting method
CN111461133B (en) Express delivery surface single item name identification method, device, equipment and storage medium
CN111062252B (en) Real-time dangerous goods semantic segmentation method, device and storage device
CN111814739B (en) Method, device, equipment and storage medium for detecting express package volume
CN110992305A (en) Package counting method and system based on deep learning and multi-target tracking technology
CN111767902A (en) Method, device and equipment for identifying dangerous goods of security check machine and storage medium
CN112819796A (en) Tobacco shred foreign matter identification method and equipment
CN113516146A (en) Data classification method, computer and readable storage medium
KR20210122429A (en) Method and System for Artificial Intelligence based Quality Inspection in Manufacturing Process using Machine Vision Deep Learning
CN111428682A (en) Express sorting method, device, equipment and storage medium
CN117214178A (en) Intelligent identification method for appearance defects of package on packaging production line
CN111985269A (en) Detection model construction method, detection device, server and medium
CN111079575B (en) Material identification method and system based on package image characteristics
CN117132540A (en) PCB defect post-processing method based on segmentation model
CN114202041A (en) Packaging material detection method and device
CN111401104B (en) Classification model training method, classification method, device, equipment and storage medium
CN114596576A (en) Image processing method and device, electronic equipment and storage medium
CN113420579A (en) Method and device for training and positioning identification code position positioning model and electronic equipment
CN114693735B (en) Video fusion method and device based on target recognition
CN114310875B (en) Crankshaft positioning identification method, device, storage medium and equipment
CN118172716B (en) Method for detecting package state of crossed belt trolley based on RGB-D image
CN117854211B (en) Target object identification method and device based on intelligent vision

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information
CB02 Change of applicant information

Address after: 214105 No. 299 Dacheng Road, Xishan District, Jiangsu, Wuxi

Applicant after: Zhongke Weizhi intelligent manufacturing technology Jiangsu Co.,Ltd.

Address before: 214105 No. 299 Dacheng Road, Xishan District, Jiangsu, Wuxi

Applicant before: ZHONGKE WEIZHI INTELLIGENT MANUFACTURING TECHNOLOGY JIANGSU Co.,Ltd.

TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20200915

Address after: 214105 No. 299 Dacheng Road, Xishan District, Jiangsu, Wuxi

Applicant after: ZHONGKE WEIZHI INTELLIGENT MANUFACTURING TECHNOLOGY JIANGSU Co.,Ltd.

Address before: Zuchongzhi road Kunshan city 215347 Suzhou City, Jiangsu province No. 1699 building 7 floor

Applicant before: KUNSHAN BRANCH, INSTITUTE OF MICROELECTRONICS OF CHINESE ACADEMY OF SCIENCES

CB02 Change of applicant information
CB02 Change of applicant information

Address after: 214105 No. 979, Antai Third Road, Xishan District, Wuxi City, Jiangsu Province

Applicant after: Zhongke Weizhi Technology Co.,Ltd.

Address before: No. 299, Dacheng Road, Xishan District, Wuxi City, Jiangsu Province

Applicant before: Zhongke Weizhi intelligent manufacturing technology Jiangsu Co.,Ltd.

GR01 Patent grant
GR01 Patent grant