CN112686930A

CN112686930A - Package sorting detection method and device, computer equipment and storage medium

Info

Publication number: CN112686930A
Application number: CN201910993445.7A
Authority: CN
Inventors: 曹文武; 朱颖; 吴登禄; 李双; 王联君
Original assignee: SF Technology Co Ltd
Current assignee: SF Technology Co Ltd; SF Tech Co Ltd
Priority date: 2019-10-18
Filing date: 2019-10-18
Publication date: 2021-04-20

Abstract

The application relates to a parcel sorting detection method, a parcel sorting detection device, computer equipment and a storage medium. The method comprises the following steps: obtaining a sorting image sequence when the packages are sorted; performing background modeling based on the sorting image sequence to obtain a package boundary frame of the fallen package; determining the overlapping area of the wrapping bounding boxes and each wrapping lattice opening and the total covering length of the wrapping bounding boxes covering the boundary line of each wrapping lattice opening; whether the parcels are successfully sorted is detected based on the overlapping area and the total covered length. By adopting the method, the working efficiency can be improved.

Description

Package sorting detection method and device, computer equipment and storage medium

Technical Field

The application relates to the field of logistics, in particular to a parcel sorting detection method, a parcel sorting detection device, computer equipment and a storage medium.

Background

In the field of logistics, parcel sorting is an important component of logistics operations. With the development of automation technology in the field of logistics, the automatic package sorting by using the mechanical arm gradually replaces the manual sorting mode. Compare in traditional manual sorting mode, the cost of labor not only can be saved in the arm letter sorting, can also accomplish the timeliness of letter sorting assurance commodity circulation fast.

However, most of the packages are currently sorted by the mechanical arm and dropped from high altitude, and due to mechanical movement errors and the like, the sorting may be inaccurate. For such abnormal situations, it is often necessary to manually perform a review process after the package is sorted, thereby reducing the work efficiency.

Disclosure of Invention

In view of the above, it is necessary to provide a parcel sorting detection method, apparatus, computer device and storage medium capable of improving work efficiency.

A package sorting detection method, the method comprising:

obtaining a sorting image sequence when the packages are sorted;

performing background modeling based on the sorting image sequence to obtain a package boundary frame of the fallen package;

determining the overlapping area of the wrapping boundary frame and each wrapping lattice and the total covering length of the boundary line of the wrapping boundary frame covering each wrapping lattice;

detecting whether the parcel is successfully sorted based on the overlap area and the total covered length.

In one embodiment, the background modeling based on the sorted image sequence to obtain the parcel bounding box of the dropped-off parcel comprises:

determining a parcel background image and a parcel foreground image from the sorted image sequence by using a frame difference method;

comparing the parcel background image with the parcel foreground image to determine a foreground target area;

and acquiring a minimum rectangular boundary frame of the foreground target area, wherein the minimum rectangular boundary frame is a package boundary frame of a falling piece package.

In one embodiment, the determining the parcel background image and the parcel foreground image from the sorted image sequence by using a frame difference method includes:

sequentially taking each image in the sorting image sequence as a first background image;

comparing the first background image with a subsequent image by using a frame difference method;

when a change region of the next image relative to the current first background image occurs and the change region is not in a preset shape, removing the current first background image from the sorted image sequence to obtain a moving image sequence;

sequentially taking each image in the moving image sequence as a second background image;

comparing the second background image with a subsequent image by using a frame difference method;

when the changing area of the latter image is determined to be in a preset shape, removing a current second background image from the moving image sequence to obtain a falling image sequence;

and selecting an image meeting the requirement from the falling image sequence as a package foreground image, and arbitrarily selecting an image from the preamble images of the falling image sequence as a package background image.

In one embodiment, the comparing the parcel background image and the parcel foreground image to determine a foreground target region comprises:

calculating the absolute value of the difference value of the pixel points corresponding to the parcel background image and the parcel foreground image;

and taking an image area formed by the pixel points with the absolute values larger than the background threshold value as a foreground target area.

In one embodiment, the determining the overlapping area of the parcel bounding box and each parcel compartment and the total covered length of the border line of the parcel bounding box covering each parcel compartment includes:

acquiring a grid distribution image, and determining the coordinates of pixel points wrapping grids according to the grid distribution image;

determining the length and the width of an overlapping area according to the coordinates of the pixel points of the wrapping border frame and the coordinates of the pixel points of each wrapping lattice;

and determining the overlapping area of the wrapping bounding box and each wrapping lattice opening and the total covering length of the boundary line of the wrapping bounding box covering each wrapping lattice opening according to the length and the width of the overlapping area.

In one embodiment, the detecting whether the parcel is successfully sorted based on the overlap area and the total covered length includes:

determining whether a parcel falls within a parcel shelf based on the overlap area and the total length of coverage;

when determining that the package does not fall into the package grid, determining that the package sorting is abnormal;

when determining that the package falls into the package grid, acquiring the grid number of the falling package grid;

when the grid number of the falling parcel grid is consistent with the sorting number, determining that the parcel sorting is successful;

and when the grid number of the falling parcel grid is inconsistent with the sorting number, determining that the parcel sorting is abnormal.

In one embodiment, determining whether a parcel falls within a parcel shelf based on the overlap area and the total length of coverage comprises:

determining that the parcel falls within a parcel shelf when the overlap area is equal to the area of the parcel bounding box;

when the overlapping area is smaller than the area of the parcel bounding box and the total length of coverage is not greater than a length threshold, determining that the parcel falls into a parcel bay;

when the overlap area is less than the area of the parcel bounding box and the total length of coverage is greater than a length threshold, determining that the parcel does not fall within a parcel shelf.

A package sorting detection apparatus, the apparatus comprising:

the acquisition module is used for acquiring a sorting image sequence during package sorting;

the modeling module is used for carrying out background modeling on the basis of the sorting image sequence and acquiring a package boundary frame of the falling piece package;

the determining module is used for determining the overlapping area of the wrapping boundary frame and each wrapping lattice and the total covering length of the boundary line of the wrapping boundary frame covering each wrapping lattice;

a detection module to detect whether the parcel was successfully sorted based on the overlap area and the total covered length.

A computer device comprising a memory storing a computer program and a processor implementing the steps of any of the above parcel sorting detection methods when the computer program is executed.

A computer-readable storage medium, having stored thereon a computer program which, when executed by a processor, carries out the steps of any of the above-mentioned parcel sorting detection methods.

According to the parcel sorting detection method, the parcel sorting detection device, the computer equipment and the storage medium, the sorting image sequence of the parcels is collected during parcel sorting, the parcel boundary frame of the fallen parcel is determined from the collected sorting image sequence by utilizing background modeling, and whether the parcels are successfully sorted is determined according to the overlapping area of the parcel boundary frame and the parcel lattice and the total length of the boundary line covering the parcel lattice, so that the sorting result of the parcels is automatically detected by utilizing a computer vision technology, manual secondary rechecking processing is not needed, and the working efficiency is improved.

Drawings

FIGS. 1a-1b illustrate an exemplary environment in which a package sorting inspection method may be implemented;

FIG. 2 is a schematic flow chart of a package sorting detection method according to one embodiment;

FIG. 3 is a schematic view of a cell distribution according to an embodiment;

FIG. 4 is a schematic illustration of a drop of parcels in one embodiment;

FIG. 5 is a flowchart illustrating the steps of obtaining a parcel bounding box for a dropped parcel based on background modeling of a sorted image sequence in one embodiment;

FIG. 6 is a block diagram of the package sorting detection apparatus according to one embodiment;

FIG. 7 is a diagram illustrating an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

The parcel sorting detection method provided by the application can be applied to the application environment shown in fig. 1a and 1 b. Fig. 1a includes an industrial personal computer 10, an image acquisition card 20, and a sorting equipment structure 100 including an industrial camera 30. The industrial personal computer 10 is connected with the image acquisition card 20 through a network, and the image acquisition card 20 is connected with the industrial camera 30 through a network. Fig. 1b is a schematic structural diagram of a sorting apparatus structure 100, which includes an industrial camera 30, an industrial camera support 40 for placing the industrial camera 30, a parcel shelf 50 for placing a parcel, a mechanical arm 80 for grabbing the parcel, and a parcel conveying platform 60 for conveying the parcel 70, wherein a barcode 701 is attached to the parcel 70, and a camera scanning support 90 for installing a camera scanning is attached to the parcel 70, and the camera scanning can be installed above the camera scanning support 90 for scanning the barcode 701 on the parcel 70 to obtain a sorting number corresponding to the parcel 70. Wherein, the industrial personal computer 10 is connected with the camera scanner and the mechanical arm 80 through a network.

Specifically, when sorting packages, the packages 70 are first placed on the package transfer platform 60. The packages are transported by the package transporting platform 60 to a place where the camera scanner can scan on the camera scanner bracket 90 and where the robotic arm 80 can grab. Then, the barcode 701 on the package 70 is scanned by a barcode scanning camera to obtain a corresponding sorting number. The scanning camera sends the scanned sorting number to the industrial personal computer 10 through the network, and the industrial personal computer 10 sends a corresponding sorting instruction to the mechanical arm 80 according to the sorting number. The robotic arm 80, in response to receiving the sorting instructions and the sorting number, picks the corresponding package 70 and begins moving toward the package pocket 50 corresponding to the sorting number. After the robotic arm 80 determines to move to the corresponding parcel bay 50, the gripped parcel 70 is released to complete the drop off of the parcel 70.

In the above parcel sorting process, the industrial camera 30 continuously performs image acquisition, and a series of acquired images are a sorting image sequence. Then, the industrial camera 30 transmits the collected sorted image sequence to the image capture card 20 connected thereto, and the collected sorted image sequence is transmitted to the industrial personal computer 10 through the image capture card 20, so that the industrial personal computer 10 can obtain the sorted image sequence to complete package sorting detection based on the sorted image sequence. Specifically, after the industrial personal computer 10 obtains the sorted image sequence, background modeling is performed based on the sorted image sequence, and a package boundary frame of the fallen package is obtained; the industrial personal computer 106 determines the overlapping area of the wrapping boundary frames and each wrapping lattice opening and the total covering length of the wrapping boundary frames covering the boundary lines of each wrapping lattice opening; the industrial personal computer 106 detects whether the parcels are successfully sorted based on the overlapping area and the total length of coverage.

In one embodiment, as shown in fig. 2, a parcel sorting detection method is provided, which is described by taking the example that the method is applied to the industrial personal computer in fig. 1, and includes the following steps:

step S202, a sorting image sequence during package sorting is obtained.

The sorting image sequence refers to a series of images acquired in the process of sorting the packages. The sorting image sequence comprises an image before the parcel reaches the drop zone, an image moving in the drop zone and an image of the drop. The image of the parcel before reaching the drop zone is an image without the mechanical arm in the image. The image moved in the drop zone refers to an image including a movement of the moving device to grab the parcel, for example, a movement of the robot arm to grab the parcel. It is understood that the image containing the robot arm grasping the package is a moving image. And the image of the falling piece is the image of the mechanical arm and the package in a separated state in the image.

Specifically, the sorted image sequence may be collected in advance and stored in a memory of an industrial personal computer, and the industrial personal computer performs package sorting detection by locally acquiring the sorted image sequence. Or a sorting image sequence acquired by an industrial personal computer in real time. I.e. when the sorting of the packages is started, the industrial camera for capturing images and the robot arm for sorting the packages are activated. And the mechanical arm grabs the packages according to the received sorting instructions. And then, after the sorting number of the package lattice of the package falling part is determined according to the code scanning result of the grabbed package, the package is moved to the package lattice corresponding to the sorting number to fall the part. In the process, the industrial camera carries out image acquisition, and a series of acquired images are sorting image sequences. For example, the industrial camera takes pictures and fetches pictures when the mechanical arm receives a sorting instruction and starts to take pictures. The industrial camera sends the collected sorting image sequence to the industrial personal computer through the image acquisition card, and the industrial personal computer acquires the sorting image sequence.

And step S204, performing background modeling based on the sorting image sequence, and acquiring a package boundary frame of the falling piece package.

The background modeling is a moving object detection method, and a background model is obtained mainly by modeling the background of an image. Once the background model is established, the current image is compared with the background model, so that the foreground target is determined according to the comparison result, and the foreground target is the moving target to be detected. Drop-off packages refer to packages that have dropped off of the sorted packages. The parcel bounding box is the parcel detected in the embodiment, and the shape and size of the parcel and the position of the parcel in the image can be represented by the bounding box.

Specifically, after the sorted image sequence is acquired, a package background image is determined from the sorted image sequence by using a background modeling method. Then, each image in the sorted image sequence is compared with the package background image respectively to determine a package foreground image. Methods of background modeling include, but are not limited to, color background modeling and average background modeling. The color background modeling method is mainly based on color values of all pixels in an image for modeling, and the color background modeling method can be determined according to the average value of the color values or other arbitrary related values according to actual conditions. For example, the average value of the color value of each pixel in each image in the collected sorted image sequence is used as the color value of the image pixel, so that the obtained image is the package background image. Then, taking each image in the sorted image sequence as a current image, and comparing the current image with the color values of the pixels at the corresponding positions on the package background image one by one, and when the color values of the pixels at the corresponding positions have large difference, determining that the area where the pixel is located is a foreground, otherwise, determining that the pixel is a background.

The average background modeling method is to model based on the pixel average value of each corresponding pixel in the image, namely, calculate the average value of the corresponding pixels of each image in the sorted image sequence, and use the image formed by the average values as the package background image. And then taking each image in the sorted image sequence as a current image to perform difference operation with pixel values of corresponding pixel points on the package background image, and comparing the obtained difference with a threshold value to determine whether the area where the pixel is located is a background or a foreground.

The background is an image area where an image is not changed from the wrapping background image, and the foreground is an image area where an image is changed from the wrapping background image. When it is determined that one image in the sorting image sequence has a changed area compared with the parcel background image, the image is the parcel foreground image. Since the condition for determining the image as the package foreground image is that the image has a variation region, after the package foreground image is determined, the determined variation region can be directly used as the region where the package exists. It is also possible to compare the parcel background image with the determined parcel foreground image again after the parcel foreground image is determined. Namely, the pixel values of the pixels at the corresponding positions between the parcel background image and the parcel foreground image are subjected to difference operation, and the variation area is determined according to the obtained difference. Then, the minimum bounding rectangle of the changed region is obtained, and since the changed region is the region where the parcel is located, the minimum bounding rectangle is actually the shape, area and position of the parcel on the image, and the minimum bounding rectangle is the parcel bounding box.

It should be understood that the background modeling described in the present embodiment is exemplified by only images including dropped pieces in the sorted image sequence. And when the sorting image sequence simultaneously comprises an image before the package reaches the drop area, an image moving in the drop area and an image of the drop, the image before the package reaches the drop area, the image moving in the drop area and the package boundary frame for obtaining the drop package need to be removed.

Step S206, determining the overlapping area of the parcel bounding box and each parcel lattice, and the total coverage length of the border line of the parcel bounding box covering each parcel lattice.

The overlapping area refers to an area of an overlapping region of the parcel border frame and the parcel lattice on the image, and may be understood as an area of the parcel lattice covered by the parcel border frame. Because sorting packages places the packages into corresponding package compartments and the image captured by the industrial camera is a flat image. Thus, when a parcel does not fall exactly into a parcel shelf, the parcel is actually above the parcel shelf. The parcel compartment blocked by the parcel is not visible from the image captured by the industrial camera, i.e., the parcel compartment is covered by the parcel bounding box. And the part of the area covered by the blocking is the overlapping area of the parcel bounding box and the parcel lattice, and the area of the overlapping area is the overlapping area. In this embodiment, the overlapping area refers to the overlapping area of the parcel bounding box and a parcel shelf. The covering length is the length covered by the wrapping border frame when the boundary line of the wrapping lattice is covered, and the total covering length is the total length of all the boundary lines covered by the wrapping border frame. For example, when two boundary lines covered by the parcel border frame are boundary lines of two different parcel compartments, the total covered length is the length obtained by adding the lengths of the two boundary lines. Similarly, the boundary line covered is the boundary line of the parcel shelf blocked by the parcel.

In one embodiment, the determining the overlapping area of the parcel bounding box and each parcel shelf, and the total covered length of the border line of the parcel bounding box covering each parcel shelf specifically includes: acquiring a grid distribution image, and determining the coordinates of pixel points wrapping grids according to the grid distribution image; determining the length and the width of the overlapping area according to the coordinates of the pixel points wrapping the boundary frame and the coordinates of the pixel points wrapping the lattice; and determining the overlapping area of the wrapping bounding box and each wrapping cell and the total covering length of the boundary line of the wrapping bounding box covering each wrapping cell according to the length and the width of the overlapping area.

Referring to the schematic diagram of the grid distribution shown in fig. 3, a grid distribution image is used to show the position distribution of each parcel grid on the image.

Specifically, in the present embodiment, the shapes of the wrapping bounding box and the wrapping lattice are both rectangular. Therefore, when determining the overlapping area and the total coverage length, the vertex coordinates of the wrapping bounding box are determined by the coordinates of each pixel point of the wrapping bounding box. And acquiring the initial coordinates and the end coordinates of the package from the vertex coordinates. Similarly, the start coordinate and the end coordinate of the parcel shelf are determined according to the vertex coordinate of the parcel bounding box. The start and end coordinates are coordinates where neither the x nor y coordinates intersect. For example, the vertex coordinates of the rectangle are (x11, y11), (x11, y12), (x12, y11), (x12, y12), and the coordinates where x and y do not intersect are (x11, y11) and (x12, y 12). Then (x11, y11) is the start coordinate, (x12, y12) is the end coordinate, or (x12, y12) is the start coordinate, (x11, y11) is the end coordinate.

After the parcel bounding box and the start coordinate and the end coordinate of each parcel lattice are obtained respectively, the length and the width of the overlapping area of the parcel bounding box and the parcel lattice can be calculated according to the start coordinate and the end coordinate. Namely, the length and the width of the parcel bounding box are calculated according to the initial coordinate and the end coordinate of the parcel bounding box. Similarly, the length and width of the parcel shelf are calculated according to the start coordinate and the end coordinate of the parcel shelf. The length is the absolute value of the difference between the x coordinate in the start coordinate and the end coordinate, and the width is the absolute value of the difference between the y coordinate in the start coordinate and the end coordinate. Then, the length of the overlapping area is obtained through calculation according to the length of the wrapping boundary frame and the length of the wrapping lattice, and the length of the overlapping area is obtained through calculation according to the width of the wrapping boundary frame and the width of the wrapping lattice.

Taking the parcel bounding box and the parcel lattice both with the coordinate with smaller x coordinate as the starting coordinate and the other coordinate as the ending coordinate, the calculation formulas of the length and the width of the overlapping area are as follows:

the length of the overlap region | + | length of the wrap bounding box | - | length of the wrap bay | - | wrap bounding box start x coordinate-end x coordinate of the wrap bay |,

the width of the overlap region | + | the width of the wrapping bounding box | + | the width of the wrapping bay | - | the initial y coordinate of the wrapping bounding box-the terminal y coordinate of the wrapping bay |.

After the length and the width of the overlapping area are obtained, the multiplication machine of the length and the width of the overlapping area is the overlapping area. And the sum of the length and the width is the length of the covered boundary line of the overlapped wrapping cell, and the sum of the lengths of all the boundary lines of the wrapping cells overlapped with the wrapping boundary frame is the total covered length. It should be understood that, for the sake of calculation, the coordinates in the present embodiment are all exemplified as being located in the first quadrant.

And step S208, detecting whether the parcels are successfully sorted or not based on the overlapping area and the total covered length.

In particular, reference is made to the schematic illustration of the package drop shown in fig. 4. And judging whether the parcel falls into the parcel grid or not according to the overlapping area and the total coverage length after the overlapping area and the total coverage length are obtained. When the package does not fall into the package grid, the package is not successfully dropped, and the package sorting abnormity can be determined. And when the package falls into the package grid, the package is indicated to successfully fall. When the packages are successfully dropped, the package sorting success can be determined.

According to the parcel sorting detection method, the sorting image sequence of the parcels is collected during parcel sorting, the parcel boundary frame of the fallen parcels is determined from the collected sorting image sequence by utilizing background modeling, and whether the parcels are successfully sorted is determined according to the overlapping area of the parcel boundary frame and the parcel lattice and the total length of the boundary line covering the parcel lattice, so that the sorting result of the parcels is automatically detected by utilizing a computer vision technology, manual secondary rechecking processing is not needed, and the working efficiency is improved.

In one embodiment, as shown in fig. 5, the step of performing background modeling based on the sorted image sequence to obtain the parcel bounding box of the dropped-off parcel comprises the steps of:

step S502, determining a parcel background image and a parcel foreground image from the sorted image sequence by using a frame difference method.

The frame difference method is also called image subtraction processing, and refers to performing difference operation on pixel values of pixel points at corresponding positions of two adjacent images, and determining the difference between the two images according to the difference. In this embodiment, the parcel background image and the parcel foreground image are determined from the sorted image sequence using a frame difference method.

Specifically, when background modeling is performed based on the sorted image sequence, images in the sorted image sequence are sequentially taken as package background images. And then comparing the parcel background image with a subsequent image corresponding to the parcel background image, and determining whether the subsequent image has image difference with the parcel background image or not through a comparison result, namely whether the subsequent image changes compared with the parcel background image or not. And when the fact that the subsequent image has a changed area is determined, and the shape of the falling parcel is determined according to the shape of the changed area, determining that the object in the changed area is the falling parcel, and determining that the subsequent image is a parcel foreground image. When the image change region does not exist or the object with the change region does not belong to a falling parcel, the subsequent image is determined not to be a parcel foreground image. And when the subsequent image is determined not to be the package foreground image, taking the image as a package background image for comparison with the corresponding subsequent image until the package background image is obtained.

The parcel background image is compared with the corresponding subsequent image, and a difference value is obtained mainly through difference value operation of pixel points of corresponding positions of the subsequent image and the parcel background image. And comparing the absolute value of the difference with a background threshold, and determining the pixel point as a foreground pixel point when the absolute value is greater than the background threshold. The method is applied to the whole image, all foreground pixel points can be obtained, and an image area formed by the foreground pixel points is a change area. The background threshold is a preset pixel value used for determining whether the pixel point is a background or a foreground.

And step S504, comparing the parcel background image with the parcel foreground image to determine a foreground target area.

Specifically, after the obtained package foreground image is obtained, the obtained package foreground image is compared with the package background image, and a change area of the package foreground image compared with the package background image is determined, wherein the change area is a foreground target area. It should be understood that since the parcel foreground image is derived from the image compared to the parcel background image, the region of variation has been determined in the comparison. In order to accelerate the detection and improve the efficiency, a changed region obtained by comparing the package foreground image with the package background image before the package foreground image can be directly obtained as a foreground target region. In this embodiment, the second recheck can be performed by the second comparison, so as to improve the accuracy of the detection.

In one embodiment, the parcel background image and the parcel foreground image are compared to obtain a foreground target area, and the foreground pixel point is determined according to the absolute value of the difference value and the background threshold value by calculating the difference value of the pixel points corresponding to the parcel foreground image and the parcel background image. And when the absolute value of the difference is larger than the background threshold, determining the pixel point corresponding to the difference as a foreground pixel point. The method is applied to the whole image, all foreground pixel points can be obtained, and an image area formed by the foreground pixel points is a change area.

Step S506, a minimum rectangular boundary box of the foreground target area is obtained, and the minimum rectangular boundary box is a package boundary box of the falling piece package.

The minimum rectangular bounding box is a minimum rectangle surrounding a foreground target area, the foreground target area is an area where the package is located, and the minimum rectangular bounding box is the package bounding box of the package.

Specifically, after the foreground target area is obtained, the smallest rectangular frame drawing is performed on the foreground target area by calling a corresponding function in the image processing tool, so that a parcel bounding frame is obtained. Image processing tools include, but are not limited to, OpenCV, MATLAB, and the like.

In one embodiment, the determining the parcel background image and the parcel foreground image from the sorted image sequence by using a frame difference method specifically includes: sequentially taking each image in the sorted image sequence as a first background image; comparing the first background image with a subsequent image by using a frame difference method; when the latter image has a change region relative to the current first background image and the change region is not in a preset shape, removing the current first background image from the sorted image sequence to obtain a moving image sequence; taking each image in the moving image sequence as a second background image in sequence; comparing the second background image with a subsequent image by using a frame difference method; when the changing area of the next image is determined to be in a preset shape, removing the current second background image from the moving image sequence to obtain a falling image sequence; and selecting an image meeting the requirement from the falling piece image sequence as a package foreground image, and arbitrarily selecting an image from the preamble images of the falling piece image sequence as a package background image.

The moving image sequence refers to a sorting image sequence with images before packages reach the drop area removed, and the drop image sequence refers to a moving image sequence with images moving in the drop area removed. For example, the sequence of sorted images includes images 1, 2, 3, 4, 5, 6. 1 and 2 are images before the parcel reaches the drop zone, 3 and 4 are images moving in the drop zone, and 5 and 6 are images of the drop. The moving image sequence should comprise 3, 4, 5, 6. The drop image sequence comprises 5, 6. The preset shape refers to the shape of the parcel, which in this embodiment is rectangular.

Specifically, taking the above-mentioned sorted image sequences 1, 2, 3, 4, 5, 6 as an example, it is assumed that the sorted image sequences are a sorting process for sorting one package. Since the industrial camera receives the sorting instruction from the mechanical arm to start collecting, the first image is the background image. Therefore, 1 is first taken as the first background image. Then, 1 and 2 are compared by the frame difference method, and if 2 is determined to have no fluctuation region, 2 is compared with 3 as the first background image. The comparison is the difference operation of the pixel points at the corresponding positions between 1 and 2, and between 2 and 3, so as to obtain the difference. And comparing the absolute value of the difference with a background threshold, and determining the pixel point as a foreground pixel point when the absolute value is greater than the background threshold. All foreground pixel points are obtained by the method, and an image area formed by the foreground pixel points is a change area. When it is determined that the image 3 has a variation region and the shape of the variation region is not a rectangle but a shape of the parcel captured by the mechanical arm, it may be determined that the image 3 is an image of the parcel captured by the mechanical arm reaching the drop area. The current first background image is removed, i.e. 1 and 2 are removed, and the remaining 3, 4, 5, 6 are moving image sequences.

Likewise, 3 is compared with 4 as the second background image. When it is determined that there is a variation region in 4, but the variation region in 4 is still the shape of the parcel grasped by the robot arm, it means that the variation region in 4 is not a predetermined rectangle. That is, 4 is compared with 5 as the second background image, and when it is determined from the comparison that 5 has a variation region including a single parcel shape and a robot arm shape, that is, the variation region has a preset rectangle, 5 is determined as the image of the drop. The current second background images 3 and 4 are removed and the remaining 5 and 6 are the drop image sequence. Then, randomly selecting an image from 1, 2, 3 and 4 as a package background image. And because the images acquired in the package dropping process may acquire images with double images, images with clearer quality are selected from 5 and 6 as package foreground images. The pixel points corresponding to the ghost images of the image are changed, so that the image with the clearest wrapping shape can be selected as a wrapping foreground image according to the pixel points.

It will be appreciated that if the sequence of captured sorted images includes a plurality of parcel sorting passes, for example 1, 2, 3, 4, 5, 6 followed by images other parcel sorted images 7, 8, 9 … … n. I.e., the first parcel is sorted and then there is an image of the sorting process for the second parcel, the third parcel … …, the nth parcel. Therefore, to prevent the detected parcel from affecting the subsequent detection, the detected image should be compared with the subsequent image as a background image. That is, after a first parcel drops, the corresponding image before the drop zone should be the image of the first parcel containing the dropped parcel, as opposed to a second parcel. For example, compare 6 as the background image to 7, determine if 7 is the image before the second parcel reaches the drop off area. When only the first package is included in 7, or a separate robotic arm and first package are included, then 7 is determined to be an image of the second package before it reaches the drop off area. Then, 7 is taken as a background image to be compared with 8, when the first parcel is determined to be contained in 8 and the mechanical arm is further involved in grabbing a second parcel, 8 is determined to be an image moving in the falling area of the second parcel, and the images before 7 and 7 are removed. In comparing 8 with 9 to determine the image of the second parcel falling piece, the subsequent processing means is the same as the processing means for sorting the image sequence only including sorting one parcel, and the details are not repeated here.

In one embodiment, detecting whether the parcel is successfully sorted based on the overlapping area and the total covered length specifically includes: determining whether the parcel falls within a parcel shelf based on the overlap area and the total length of coverage; when determining that the packages do not fall into the package grid, determining that the package sorting is abnormal; when determining that the package falls into the package grid, acquiring the grid number of the falling package grid; when the grid number of the falling package grid is consistent with the sorting number, determining that the package sorting is successful; and when the grid number of the falling package grid is inconsistent with the sorting number, determining that the package sorting is abnormal.

Wherein, the gridwork bead number is the only number that sets up for wrapping up the gridwork bead in advance, and this number can be according to the coordinate generation of each parcel gridwork bead pixel. The sort number is the bin number of the package bin into which the package should fall. When the actual number of the falling grid is consistent with the sorting number, the package is shown to fall into the correct grid, and when the actual number of the falling grid is not consistent with the sorting number, the package is not shown to fall into the correct grid.

Specifically, after confirming that the parcel falls into the parcel check according to the overlapping area and the total length of coverage, the check mouth number of the parcel check that the parcel falls into can also be further obtained, and whether the parcel has a wrong check mouth is judged according to the check mouth number of the parcel. If the grid number and the coordinates of the pixel points of each wrapped grid are mapped in advance, the grid number of the wrapped grid can be directly obtained according to the mapping relation. Or converting the coordinates of the pixel points wrapping the grid to obtain the corresponding grid number. And when the grid number of the package grid falling into the package is inconsistent with the sorting number, determining that the package is in a wrong grid, and similarly, determining that the package is abnormal in sorting. And if the lattice is not missed, the sorting success is determined.

In addition, when the parcel sorting is abnormal, corresponding abnormal information is stored and returned to sorting workers, and the corresponding abnormal information is used for reminding the sorting workers to perform manual reinspection in time. The abnormal information includes an image corresponding to the sorting abnormality, namely a package foreground image, sorting time of the package with the sorting abnormality, the sorting time can be obtained according to the corresponding image, and a grid number related to the package with the sorting abnormality, wherein the related grid number is a grid number of a package grid with an overlapping area with the package. And the abnormal information can be saved by calling a function corresponding to the image processing tool. For example, the corresponding abnormal image can be saved by the inwrite function of the OpenCV tool, and the localtime function of the OpenCV tool can record the time, the cell number, and the like.

In one embodiment, determining whether a parcel falls within a parcel shelf based on the overlap area and the total length of coverage comprises: when the overlapping area is equal to the area of the parcel bounding box, determining that the parcel falls into a parcel shelf; when the overlapping area is smaller than the area of the parcel bounding box and the total covering length is not larger than the length threshold, determining that the parcel falls into a parcel lattice; and when the overlapping area is smaller than the area of the parcel bounding box and the total covered length is larger than the length threshold value, determining that the parcel does not fall into the parcel lattice.

Specifically, the area of overlap of the parcel bounding box with the parcel shelf is unlikely to exceed the area of the parcel bounding box by the maximum, and when a parcel falls into a parcel shelf, the area of overlap should be the area of the parcel. Thus, when the overlap area is determined to be equal to the area of the parcel bounding box, it indicates that the parcel falls within the parcel shelf. And when the overlapping area is smaller than the area of the parcel bounding box, whether the parcel falls into a parcel lattice needs to be further determined according to the total length of the coverage. Because the collected image is a plane image, when the package falls into the package lattice in an inclined state and is in the condition of just blocking the boundary line of the package lattice, the calculated overlapping area is smaller than the area of the package boundary frame to a certain extent, but the state of the package falls into the package lattice at the moment. Therefore, a further judgment is needed according to the total coverage length. And comparing whether the total covered length is greater than the length threshold value to determine whether the total covered length falls into the grid or not by acquiring the length threshold value. The length threshold is predetermined based on the size of the actual parcel shelf and the largest parcel actually sorted. For example, the length threshold should be set to 400mm when the maximum parcel size is 324mm by 229mm and the parcel shelf size is 800mm by 60mm, as tested.

When the total coverage length is less than or equal to the length threshold, the coverage rate of the boundary line does not reach the step of supporting the parcel, so that the parcel can be considered to have fallen into the parcel grid. And when the total coverage length is greater than the length threshold, indicating that the coverage rate of the dividing line has reached a point where the parcel can be supported, it can be determined that the parcel does not fall within the parcel shelf. In this embodiment, whether sorting is successful is further determined by combining the coverage degree of the boundary, so that the detection accuracy can be improved.

It should be understood that although the steps in the flowcharts of fig. 2 and 5 are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2 and 5 may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least some of the sub-steps or stages of other steps.

In one embodiment, as shown in fig. 6, there is provided a package sorting detection apparatus including: an acquisition module 602, a modeling module 604, a determination module 606, and a detection module 608, wherein:

an obtaining module 602, configured to obtain a sorting image sequence when a package is sorted.

And the modeling module 604 is configured to perform background modeling based on the sorted image sequence to obtain a parcel bounding box of the dropped parcel.

A determining module 606, configured to determine an overlapping area of the parcel bounding box and each parcel bay, and a total covered length of a boundary line of the parcel bounding box covering each parcel bay.

A detection module 608 that detects whether the parcel was successfully sorted based on the overlap area and the total length of coverage.

In one embodiment, the modeling module 604 is further configured to determine a parcel background image and a parcel foreground image from the sorted image sequence using frame differencing; comparing the parcel background image with the parcel foreground image to determine a foreground target area; and acquiring a minimum rectangular boundary frame of the foreground target area, wherein the minimum rectangular boundary frame is a package boundary frame of a falling piece package.

In one embodiment, the modeling module 604 is further configured to take each image in the sequence of sorted images as a first background image in turn; comparing the first background image with a subsequent image by using a frame difference method; when the latter image has a change region relative to the current first background image and the change region is not in a preset shape, removing the current first background image from the sorted image sequence to obtain a moving image sequence; taking each image in the moving image sequence as a second background image in sequence; comparing the second background image with a subsequent image by using a frame difference method; when the changing area of the next image is determined to be in a preset shape, removing the current second background image from the moving image sequence to obtain a falling image sequence; and selecting an image meeting the requirement from the falling piece image sequence as a package foreground image, and arbitrarily selecting an image from the preamble images of the falling piece image sequence as a package background image.

In one embodiment, the modeling module 604 is further configured to calculate an absolute value of a difference between corresponding pixel points of the parcel background image and the parcel foreground image; and taking an image area formed by pixel points with absolute values larger than the background threshold value as a foreground target area.

In one embodiment, the determining module 606 is further configured to obtain a cell distribution image, and determine coordinates of pixel points wrapping each cell according to the cell distribution image; determining the length and the width of the overlapping area according to the coordinates of the pixel points wrapping the boundary frame and the coordinates of the pixel points wrapping the lattice; and determining the overlapping area of the wrapping bounding box and each wrapping cell and the total covering length of the boundary line of the wrapping bounding box covering each wrapping cell according to the length and the width of the overlapping area.

In one embodiment, the detection module 608 is further configured to determine whether the parcel falls within a parcel shelf based on the overlap area and the total length of coverage; when determining that the packages do not fall into the package grid, determining that the package sorting is abnormal; when determining that the package falls into the package grid, acquiring the grid number of the falling package grid; when the grid number of the falling package grid is consistent with the sorting number, determining that the package sorting is successful; and when the grid number of the falling package grid is inconsistent with the sorting number, determining that the package sorting is abnormal.

In one embodiment, the detection module 608 is further configured to determine that the parcel falls within the parcel compartment when the overlap area is equal to the area of the parcel bounding box; when the overlapping area is smaller than the area of the parcel bounding box and the total covering length is not larger than the length threshold, determining that the parcel falls into a parcel lattice; and when the overlapping area is smaller than the area of the parcel bounding box and the total covered length is larger than the length threshold value, determining that the parcel does not fall into the parcel lattice.

For the specific definition of the package sorting detection device, reference may be made to the above definition of the package sorting detection method, which is not described herein again. The modules in the package sorting and detecting device can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, and the computer device may be an industrial personal computer, and the internal structure diagram of the computer device may be as shown in fig. 7. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a package sorting detection method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.

Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:

obtaining a sorting image sequence when the packages are sorted;

determining the overlapping area of the wrapping bounding boxes and each wrapping lattice opening and the total covering length of the wrapping bounding boxes covering the boundary line of each wrapping lattice opening;

whether the parcels are successfully sorted is detected based on the overlapping area and the total covered length.

In one embodiment, the processor, when executing the computer program, further performs the steps of: determining a parcel background image and a parcel foreground image from the sorted image sequence by using a frame difference method; comparing the parcel background image with the parcel foreground image to determine a foreground target area; and acquiring a minimum rectangular boundary frame of the foreground target area, wherein the minimum rectangular boundary frame is a package boundary frame of a falling piece package.

In one embodiment, the processor, when executing the computer program, further performs the steps of: sequentially taking each image in the sorted image sequence as a first background image; comparing the first background image with a subsequent image by using a frame difference method; when the latter image has a change region relative to the current first background image and the change region is not in a preset shape, removing the current first background image from the sorted image sequence to obtain a moving image sequence; taking each image in the moving image sequence as a second background image in sequence; comparing the second background image with a subsequent image by using a frame difference method; when the changing area of the next image is determined to be in a preset shape, removing the current second background image from the moving image sequence to obtain a falling image sequence; and selecting an image meeting the requirement from the falling piece image sequence as a package foreground image, and arbitrarily selecting an image from the preamble images of the falling piece image sequence as a package background image.

In one embodiment, the processor, when executing the computer program, further performs the steps of: calculating the absolute value of the difference value of the pixel points corresponding to the parcel background image and the parcel foreground image; and taking an image area formed by pixel points with absolute values larger than the background threshold value as a foreground target area.

In one embodiment, the processor, when executing the computer program, further performs the steps of: acquiring a grid distribution image, and determining the coordinates of pixel points wrapping grids according to the grid distribution image; determining the length and the width of the overlapping area according to the coordinates of the pixel points wrapping the boundary frame and the coordinates of the pixel points wrapping the lattice; and determining the overlapping area of the wrapping bounding box and each wrapping cell and the total covering length of the boundary line of the wrapping bounding box covering each wrapping cell according to the length and the width of the overlapping area.

In one embodiment, the processor, when executing the computer program, further performs the steps of: determining whether the parcel falls within a parcel shelf based on the overlap area and the total length of coverage; when determining that the packages do not fall into the package grid, determining that the package sorting is abnormal; when determining that the package falls into the package grid, acquiring the grid number of the falling package grid; when the grid number of the falling package grid is consistent with the sorting number, determining that the package sorting is successful; and when the grid number of the falling package grid is inconsistent with the sorting number, determining that the package sorting is abnormal.

In one embodiment, the processor, when executing the computer program, further performs the steps of: when the overlapping area is equal to the area of the parcel bounding box, determining that the parcel falls into a parcel shelf; when the overlapping area is smaller than the area of the parcel bounding box and the total covering length is not larger than the length threshold, determining that the parcel falls into a parcel lattice; and when the overlapping area is smaller than the area of the parcel bounding box and the total covered length is larger than the length threshold value, determining that the parcel does not fall into the parcel lattice.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:

obtaining a sorting image sequence when the packages are sorted;

In one embodiment, the computer program when executed by the processor further performs the steps of: determining a parcel background image and a parcel foreground image from the sorted image sequence by using a frame difference method; comparing the parcel background image with the parcel foreground image to determine a foreground target area; and acquiring a minimum rectangular boundary frame of the foreground target area, wherein the minimum rectangular boundary frame is a package boundary frame of a falling piece package.

In one embodiment, the computer program when executed by the processor further performs the steps of: sequentially taking each image in the sorted image sequence as a first background image; comparing the first background image with a subsequent image by using a frame difference method; when the latter image has a change region relative to the current first background image and the change region is not in a preset shape, removing the current first background image from the sorted image sequence to obtain a moving image sequence; taking each image in the moving image sequence as a second background image in sequence; comparing the second background image with a subsequent image by using a frame difference method; when the changing area of the next image is determined to be in a preset shape, removing the current second background image from the moving image sequence to obtain a falling image sequence; and selecting an image meeting the requirement from the falling piece image sequence as a package foreground image, and arbitrarily selecting an image from the preamble images of the falling piece image sequence as a package background image.

In one embodiment, the computer program when executed by the processor further performs the steps of: calculating the absolute value of the difference value of the pixel points corresponding to the parcel background image and the parcel foreground image; and taking an image area formed by pixel points with absolute values larger than the background threshold value as a foreground target area.

In one embodiment, the computer program when executed by the processor further performs the steps of: acquiring a grid distribution image, and determining the coordinates of pixel points wrapping grids according to the grid distribution image; determining the length and the width of the overlapping area according to the coordinates of the pixel points wrapping the boundary frame and the coordinates of the pixel points wrapping the lattice; and determining the overlapping area of the wrapping bounding box and each wrapping cell and the total covering length of the boundary line of the wrapping bounding box covering each wrapping cell according to the length and the width of the overlapping area.

In one embodiment, the computer program when executed by the processor further performs the steps of: determining whether the parcel falls within a parcel shelf based on the overlap area and the total length of coverage; when determining that the packages do not fall into the package grid, determining that the package sorting is abnormal; when determining that the package falls into the package grid, acquiring the grid number of the falling package grid; when the grid number of the falling package grid is consistent with the sorting number, determining that the package sorting is successful; and when the grid number of the falling package grid is inconsistent with the sorting number, determining that the package sorting is abnormal.

In one embodiment, the computer program when executed by the processor further performs the steps of: when the overlapping area is equal to the area of the parcel bounding box, determining that the parcel falls into a parcel shelf; when the overlapping area is smaller than the area of the parcel bounding box and the total covering length is not larger than the length threshold, determining that the parcel falls into a parcel lattice; and when the overlapping area is smaller than the area of the parcel bounding box and the total covered length is larger than the length threshold value, determining that the parcel does not fall into the parcel lattice.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A package sorting detection method, the method comprising:

obtaining a sorting image sequence when the packages are sorted;

2. The method of claim 1, wherein the background modeling based on the sequence of sorted images to obtain a parcel bounding box for a dropped parcel comprises:

3. The method of claim 2, wherein determining a parcel background image and a parcel foreground image from the sequence of sorted images using frame differencing comprises:

4. The method of claim 2, wherein said comparing said parcel background image and said parcel foreground image to determine a foreground target region comprises:

5. The method of claim 1, wherein determining the overlapping area of the parcel bounding box with each of the parcel compartments and the total length of coverage of the bounding line of the parcel bounding box over each of the parcel compartments comprises:

6. The method of claim 1, wherein said detecting whether a parcel is successfully sorted based on said overlap area and said total length of coverage comprises:

7. The method of claim 6, wherein determining whether a parcel falls within a parcel shelf based on the overlap area and the total length of coverage comprises:

8. A package sorting detection apparatus, the apparatus comprising:

9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.