WO2021190595A1

WO2021190595A1 - Parcel detection method, device, computing apparatus, logistics system, and storage medium

Info

Publication number: WO2021190595A1
Application number: PCT/CN2021/082964
Authority: WO
Inventors: 顾睿; 邓志辉
Original assignee: 杭州海康机器人技术有限公司
Priority date: 2020-03-25
Filing date: 2021-03-25
Publication date: 2021-09-30
Also published as: CN113449532A; CN113449532B

Abstract

A parcel detection method, a device, a computing apparatus, a logistics system, and a storage medium, capable of automatically associating barcode information with a parcel. The method comprises: when a parcel passes through a detection region (S1) on a conveyor belt (110), identifying an outline of a designated region on the parcel and acquiring a detected position of the outline in the detection region (S1); predicting, according to the detected position of the outline in the detection region (S1) and a conveying speed of the conveyor belt (110), a code reading position of the outline in an imaging region of an image coordinate system of a code reading camera (130) and a first image marker corresponding to the code reading position, wherein the code reading camera (130) is downstream with respect to the detection region (S1), and the first image marker is used to mark a parcel image captured by the code reading camera (130) when the outline reaches the code reading position; upon acquisition of the parcel image corresponding to the first image marker, performing barcode identification on the parcel image; when a barcode is identified from the parcel image, performing position matching of the code reading position and the identified barcode; and if the code reading position matches the barcode, associating the matching barcode with the parcel.

Description

Method, device, computing equipment, logistics system and storage medium for detecting packages

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office with the application number 202010216758.4 on March 25, 2020, with the title of "Method, Apparatus, Computing Equipment, Logistics System and Storage Medium for Parcel Detection", and its entire contents Incorporated in this application by reference.

Technical field

This application relates to the field of logistics automation technology, and in particular to a method, device, computing device, logistics system, and storage medium for detecting packages.

Background technique

At present, in the application scenario of logistics, the package on the conveyor belt needs to be tested for attributes, for example, the size, volume, face list, barcode and other attributes of the package are tested. Devices that detect different properties of the package (such as code reading cameras and depth cameras, etc.) can be distributed at different locations on the conveyor belt.

However, when multiple packages are simultaneously transmitted on the conveyor belt, that is, when multiple packages appear side by side or staggered back and forth in the field of view of the barcode reading camera, the barcode information of the multiple packages is likely to be confused, and the barcode information cannot be confused with the barcode information. The other attributes of the package are correctly associated.

Therefore, how to accurately associate the barcode information with the package and its attributes is a technical problem that needs to be resolved.

Summary of the invention

This application proposes a method, device, computing device, logistics system, and storage medium for detecting packages, which can automatically associate barcode information with packages and their attributes.

According to one aspect of the present application, a method for detecting packages is provided, including:

When a package on a conveyor belt passes through a detection area, identifying the contour of a designated area on the package, and identifying the detection position of the contour in the detection area, the designated area including the barcode of the package;

According to the detection position of the contour in the detection area and the conveying speed of the conveyor belt, it is predicted that the reading position of the contour in the imaging area of the image coordinate system of the code reading camera and the first corresponding to the reading position An image identification, the code reading camera is located downstream of the detection area, and the first image identification is used to identify a package image taken by the code reading camera when the outline reaches the barcode reading position;

When the package image corresponding to the first image identifier is acquired, perform barcode recognition on the package image;

When the barcode in the package image is recognized, the barcode reading position is matched with the recognized barcode;

When the barcode reading position matches the barcode, the matched barcode is associated with the package.

In some embodiments, the detection position of the contour in the detection area is the coordinates of the contour in the first world coordinate system; the detection position according to the contour in the detection area and the The transmission speed of the conveyor belt, predicting the code reading position of the contour in the imaging area of the image coordinate system of the code reading camera and the first image identification corresponding to the code reading position, includes:

When the detection position is identified, the second image identifier of the image frame collected by the barcode reading camera at the current moment is determined, and the second image identifier is the frame number or the time stamp;

Obtain the moving distance of the package in a single collection cycle of the barcode reading camera;

Determine the offset position of the contour in the first world coordinate system according to the parameters of the code reading camera, the detection position of the contour in the detection area, and the conveying speed of the conveyor belt. It is satisfied that when the contour is at the offset position, the projection position of the contour in the image coordinate system of the code reading camera is in the imaging area;

When the contour is at the offset position, the projection position of the contour in the image coordinate system of the code reading camera is taken as the code reading position;

Calculate the difference between the offset position and the detection position, and according to the number of the movement distances included in the difference, determine that the barcode reading camera reaches the barcode reading when the package is from the detection position The number of image frames taken before the location;

According to the second image identifier and the number of image frames, the first image identifier corresponding to the code reading position is determined.

In some embodiments, the method of detecting the package further includes:

When the package on the conveyor belt passes through the detection area, the target attribute of the package is detected, and the target attribute includes at least one of the volume of the package, the size of the package, the quality of the package, and the face sheet of the package;

When the barcode reading position matches the barcode, the matched barcode is associated with the target attribute.

In some embodiments, the position matching the barcode reading position with the barcode includes:

Determining the barcode area of the barcode in the package image;

Determining whether the barcode area belongs to the area corresponding to the barcode reading position;

When at least a part of the barcode area belongs to the area corresponding to the barcode reading position, determining the position match between the barcode and the barcode reading position;

When the barcode area is outside the area corresponding to the barcode reading position, it is determined that the barcode and the barcode reading position do not match.

In some embodiments, the method of detecting the package further includes:

Acquiring a panoramic image of the extended processing area of the conveyor belt, wherein the extended processing area is located downstream of the code reading field in the conveying direction of the conveyor belt;

According to the detection position and the transmission speed, continuously update the predicted position of the package in the extended processing area over time, and add a tracking frame to the panoramic image according to the predicted position;

Wherein, when the package is associated with a barcode, the tracking frame is presented in a first color, and when the package is not associated with a barcode, the tracking frame is presented in a second color.

In some embodiments, when the package on the conveyor belt passes through the detection area, recognizing the contour of the designated area on the package and recognizing the detection position of the contour in the detection area includes:

When the package on the conveyor belt passes through the detection area, the depth image of the package is acquired; according to the depth image, the contour of the upper surface of the package and the detection position of the upper surface in the first world coordinate system are determined, wherein the The upper surface is the designated area; or

When the package on the conveyor belt passes through the detection area, obtain the grayscale image and depth image of the package; determine the contour of the single area of the package in the grayscale image; according to the surface of the package in the grayscale image The contour of a single area determines the first depth area in the depth image corresponding to the single area of the surface; according to the first depth area, the detection position of the contour of the single area of the surface in the first world coordinate system is determined, where The single surface area is the designated area.

In some embodiments, the method of detecting the package further includes:

Acquiring the first world coordinate system established according to the first calibration disk, the first calibration disk being placed on the conveyor belt and in the field of view of the depth camera;

Calibrate the external parameters of the depth camera according to the first world coordinate system and the image of the first calibration disk taken by the depth camera to obtain the first mapping relationship between the depth camera coordinate system and the first world coordinate system;

Acquiring a second world coordinate system established according to a second calibration disk, the second calibration disk being placed on a conveyor belt and in the code reading field of view of the code reading camera;

Calibrate the external parameters of the code reading camera according to the second world coordinate system and the image of the second calibration disk taken by the code reading camera to obtain a second mapping relationship between the code reading camera coordinate system and the second world coordinate system;

Determine the third mapping relationship between the first world coordinate system and the second world coordinate system;

According to the internal parameters of the code-reading camera, the fourth mapping relationship between the coordinate system of the code-reading camera and the image coordinate system of the code-reading camera is determined.

In some embodiments, the determining the contour of the upper surface of the package and the detection position of the upper surface in the first world coordinate system according to the depth image includes:

According to the depth image, determine the three-dimensional model of the package; according to the coordinates of the three-dimensional model in the depth camera coordinate system and the first mapping relationship, determine the contour of the upper surface of the package and the coordinates of the upper surface in the first world The detection position in the system, the detection position is represented by the coordinates of at least three vertices of the upper surface of the package in the first world coordinate system; or

Acquiring a grayscale image corresponding to the depth image;

Determining the contour of the upper surface of the package in the grayscale image;

Determining a second depth area corresponding to the upper surface in the depth image according to the contour of the upper surface of the package in the grayscale image, to obtain at least three vertices of the second depth area;

Determining the coordinates of at least three vertices of the second depth region in the depth camera coordinate system;

According to the coordinates of the at least three vertices of the second depth region in the depth camera coordinate system and the first mapping relationship, the detection position of the upper surface of the package in the first world coordinate system is determined, and the detection position It is represented by the coordinates of at least three vertices of the second depth region in the first world coordinate system.

According to one aspect of the present application, a device for detecting packages is provided, including:

The detection unit is configured to identify the contour of a designated area on the package when the package passes through the detection area on the conveyor belt, and identify the detection position of the contour in the detection area, and the designated area includes the barcode of the package;

A predicting unit for predicting the barcode reading position and the barcode reading of the contour in the imaging area of the image coordinate system of the barcode reading camera based on the detection position of the contour in the detection area and the conveying speed of the conveyor belt A first image identifier corresponding to a position, the code reading camera is located downstream of the detection area, and the first image identifier is used to identify a package image taken by the code reading camera when the outline reaches the code reading position;

The barcode recognition unit is configured to perform barcode recognition on the package image when the package image corresponding to the first image identifier is obtained;

When the barcode recognition unit recognizes the barcode in the package image, the matching unit is used to match the barcode reading position with the recognized barcode;

When the matching unit determines that the barcode reading position matches the barcode, the associating unit is used to associate the matched barcode with the package.

According to one aspect of the present application, there is provided a computing device, including: a memory; a processor; a program, stored in the memory and configured to be executed by the processor, the program including a package for executing the inspection package according to the present application Method of instruction.

According to one aspect of the present application, there is provided a storage medium storing a program, the program including instructions that, when executed by a computing device, cause the computing device to execute the method for detecting packages according to the present application.

According to one aspect of the present application, a logistics system is provided, which includes: a computing device; a conveyor belt; a depth camera; and a code reading camera.

In summary, according to the package detection solution of the present application, when the detection position of the package is detected, the code reading position of the designated area of the package in the imaging area and the first image identification corresponding to the code reading position can be predicted. Here, the barcode reading position can be considered as the projection position of the designated area of the package in the image coordinate system when the package image corresponding to the first image identifier is acquired. On this basis, according to the package detection solution of the present application, by matching the position of the barcode and the barcode reading position in the package image, the association relationship between the package and the barcode can be determined, and further, the barcode can be associated with the attributes of the package.

Description of the drawings

In order to explain the embodiments of the present application and the technical solutions of the prior art more clearly, the following briefly introduces the drawings that need to be used in the embodiments and the prior art. Obviously, the drawings in the following description are merely the present invention. For some embodiments of the application, those of ordinary skill in the art can also obtain other drawings based on these drawings.

Figure 1 shows a schematic diagram of a logistics system according to some embodiments of the present application;

FIG. 2 shows a flowchart of a method 200 for detecting a package according to some embodiments of the present application;

FIG. 3 shows a flowchart of a method 300 for detecting a package according to some embodiments of the present application;

Figure 4 shows a schematic diagram of a coordinate system in a logistics system according to some embodiments of the present application;

FIG. 5 shows a flowchart of a method 500 for determining a detection position according to some embodiments of the present application;

FIG. 6 shows a flowchart of a method 600 for determining a detection position according to some embodiments of the present application;

FIG. 7 shows a flowchart of a method 700 for determining a detection position corresponding to the upper surface of a package according to some embodiments of the present application;

Fig. 8 shows a flowchart of a method 800 for determining a detection position corresponding to the upper surface of a package according to some embodiments of the present application;

FIG. 9 shows a flowchart of a method 900 for predicting a barcode reading position according to some embodiments of the present application;

FIG. 10A shows a schematic diagram of a conveyor belt placed with a package that does not enter the field of view of the depth camera 120; FIG.

FIG. 10B shows the detection position of the package B3 in the detection area determined by the computing device 140;

FIG. 10C shows a schematic diagram of a target location of the package B3 predicted by the computing device 140;

FIG. 10D shows a schematic diagram of the projection of 4 vertices to the image coordinate system;

FIG. 10E shows the projection area in the image coordinate system when the package is at the target position in FIG. 10C;

FIG. 11 shows a flowchart of a method 1100 for position matching a barcode and a barcode reading position according to some embodiments of the present application;

Figure 12A shows a package image according to some embodiments of the present application;

Figure 12B shows a package image according to some embodiments of the present application;

Figure 13A shows a schematic diagram of a logistics system according to some embodiments of the present application;

FIG. 13B shows a schematic diagram of a panoramic image according to some embodiments of the present application;

Figure 14 shows a schematic diagram of a logistics system according to some embodiments of the present application;

FIG. 15 shows a schematic diagram of an apparatus 1500 for detecting packages according to some embodiments of the present application;

FIG. 16 shows a schematic diagram of an apparatus 1600 for detecting a package according to some embodiments of the present application;

Figure 17 shows a schematic diagram of a computing device according to some embodiments of the present application.

Detailed ways

In order to make the purpose, technical solutions, and advantages of the present application clearer, the following further describes the present application in detail with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in this application fall within the protection scope of this application.

Fig. 1 shows a schematic diagram of a logistics system according to some embodiments of the present application. As shown in FIG. 1, the logistics system may include a conveyor belt 110, a depth camera 120, a code reading camera 130 and a computing device 140.

The driving belt 110 conveys the packages according to the conveying direction of the conveying belt (for example, the direction from right to left in FIG. 1 ).

The depth camera 120 shown in FIG. 1 is, for example, a structured light camera. The structured light camera may include a laser emission module 121 and an image acquisition module 122. The field of view V1 of the image acquisition module 122 can cover the detection area S1 on the conveyor belt 110. In addition, the depth camera 120 may also use a Time of Flight (ToF, abbreviated as ToF) camera, a binocular vision (Stereo) camera, etc. The depth camera 120 can collect images of the package passing through the field of view V1, and output a sequence of image frames to the computing device 140 in real time.

The computing device 140 may be, for example, a server, a notebook computer, a tablet computer, or a handheld business communication device. The computing device 140 can build a three-dimensional model of the package according to the sequence of image frames from the depth camera 120. In this way, the computing device 140 can detect the target attribute of the package, for example, determine the size of the package or the volume of the package. In addition, the computing device 140 may determine the detection location of the package on the conveyor belt 110 in the detection area S1 at the moment when the target attribute is detected, that is, determine the actual location of the package at the current moment. Here, the detection position wrapped in the detection area S1 can be represented by, for example, the coordinates of the 4 vertices on the upper surface of the package.

In the conveying direction of the conveyor belt 110, the code reading camera 130 is downstream of the depth camera 120. The barcode reading field V2 of the barcode reading camera 130 covers the barcode identification area S2 on the conveyor belt 110. The code reading camera 130 may be an industrial camera with an image capturing function, or a smart camera integrated with image capturing and image processing functions. The code reading camera 130 may output image frames to the computing device 140. The computing device 140 can perform barcode recognition on the image frame from the barcode reading camera 130. Here, the computing device 140 can detect one-dimensional barcodes and/or two-dimensional codes.

In some embodiments, the computing device 140 may establish an association relationship between the barcode information and the target attribute. The manner of establishing an association relationship will be described below with reference to FIG. 2.

FIG. 2 shows a flowchart of a method 200 for detecting a package according to some embodiments of the present application. The method 200 may be executed by the computing device 140.

In step S201, when the package on the conveyor belt passes through the detection area, the contour of the designated area on the package is recognized, and the detection position of the contour in the detection area is recognized. The designated area includes the barcode of the package, for example, the upper surface of the package or the surface area of the package. In step S201, for example, the coordinates of at least three vertices in the designated area may be determined, and the coordinates of the at least three vertices may be used to indicate the detection position of the contour in the detection area.

In one implementation, when the package passes through the detection area on the conveyor belt, after the contour of the designated area is recognized, the coordinates of multiple vertices of the designated area can be obtained to indicate the detection position of the contour in the detection area.

In step S202, according to the detection position of the contour in the detection area and the conveying speed of the conveyor belt, the code reading position of the contour in the imaging area of the image coordinate system of the code reading camera and the first image identification corresponding to the code reading position are predicted. The code reading camera 130 is located downstream of the detection area. The first image identification is used to identify the package image taken by the barcode reading camera 130 when the outline reaches the barcode reading position. In some embodiments, the first image identifier is, for example, a frame number or a time stamp. The code reading position of the contour refers to the coordinate position of the contour in the image coordinate system of the code reading camera 130 when at least a part of the contour is in the imaging area.

The target attribute includes at least one of the following: package volume and package size. For example, in step S201, the target attribute of the package may be determined according to the sequence of image frames collected by the depth camera 120. The depth camera 120 is, for example, a line structured light camera. When the online structured light camera completes the scanning of a package, step S201 can generate a three-dimensional model of the package according to the scanned image frame sequence. In this way, in step S201, the target attribute of the package can be determined according to the three-dimensional model.

In an optional embodiment, taking the designated area on the package as the upper surface of the package as an example, the detection position of the upper surface of the package in the detection area on the conveyor belt 110 can be determined. For example, the coordinates of 4 vertices on the upper surface of the package can be determined, and the coordinates of the 4 vertices can be used to indicate the detection position of the package in the detection area. Furthermore, the detection position can be used as a starting point, and at least one code reading position can be determined according to the conveyor speed. The reading position can be indicated by the coordinates of the 4 vertices on the upper surface of the package. The coordinates of the 4 vertices can define a rectangular area. The rectangular area corresponds to the upper surface of the package. At least a part of the rectangular area is in the reading field of view.

In step S203, when the package image corresponding to the first image identifier is acquired, barcode recognition is performed on the package image.

When the barcode in the package image is identified in step S203, the method 200 may execute step S204 to position the identified barcode with the barcode reading position. Here, step S204 may perform position matching on the barcode and the barcode reading position in the same coordinate system (for example, the image coordinate system of the barcode reading camera 130). In a scenario where multiple packages are placed side by side or staggered back and forth on the conveyor belt, step S203 may identify multiple barcodes in the package image, such as barcode C1, barcode C2, and barcode C3. For the barcode reading position of the package B1, in step S204, when at least a part of the barcode area corresponding to the barcode C1 belongs to the designated area of the package B1, the position matching between the barcode C1 and the barcode reading position corresponding to the package B1 may be determined when the image of the package is projected. . That is, in step S204, it can be determined that when the outline of the designated area on the package B1 reaches the barcode reading position, the designated area of the package B1 is projected in the package image, and further, if it is determined that at least a part of the barcode area corresponding to the barcode C1 belongs to the projection area , Then it can be determined that the barcode C1 matches the barcode reading position corresponding to the package B1. Otherwise, it is determined that the position of the barcode C1 and the corresponding barcode reading position of the package B1 do not match. In the same way, it can be judged whether the barcode C2 and the corresponding reading position of the package B1 match, and whether the barcode C3 and the corresponding reading position of the package B1 match. Based on this, it is possible to determine the barcode that matches the position between the barcode reading positions corresponding to the package B1.

In one embodiment, if the first image identifier is a frame number, an image with the frame number in the image captured by the code reading camera 130 can be acquired as the package image corresponding to the first image identifier. If the first image identifier is a timestamp, an image with the timestamp in the image taken by the code reading camera 130 can be acquired as the package image corresponding to the first image identifier.

In step S205, when the barcode reading position matches the barcode, associate the matched barcode with the package. If a barcode is identified in step S204, the barcode matched in step S205 is the barcode identified in step S204. If multiple barcodes are identified in step S204, the barcode matched in step S205 may be one of the multiple barcodes identified in step S204.

In summary, according to the package detection method 200 of the present application, when the detection position of the package is detected, the code reading position of the designated area of the package in the imaging area and the first image identification corresponding to the code reading position can be predicted. Here, the barcode reading position can be considered as the projection position of the designated area of the package in the image coordinate system of the barcode reading camera when the package image corresponding to the first image identifier is acquired. On this basis, the package detection method 200 according to the present application can determine the association relationship between the package and the barcode by matching the position of the barcode and the barcode reading position in the package image. In addition, after the association relationship between the package and the barcode is determined, the attributes of the package can also be obtained, and the attributes of the package can be associated with the barcode. The attributes of the package can be detected in advance.

In one embodiment, after the computing device 140 obtains the sequence of image frames output by the depth camera 120 and recognizes the package, the identification of each package can be generated. Furthermore, after the detection position is determined, and the code reading position and the first image identification are predicted, the correspondence relationship between the identification of the package, the detection position, the reading position and the first image identification can be recorded. Subsequently, when the package image corresponding to the first image identification is obtained, the barcode is recognized and the barcode reading position matches the barcode, the identification of the package corresponding to the barcode reading position can be determined, and then the matched barcode is corresponding to the determined identification The packages are associated.

FIG. 3 shows a flowchart of a method 300 for detecting a package according to some embodiments of the present application. The method 300 may be executed by the computing device 140.

In step S301, the first world coordinate system established according to the first calibration plate is acquired. The first calibration disk is placed on the conveyor belt 110 and is in the field of view of the depth camera 120. The first calibration board is, for example, a checkerboard calibration board. As shown in Figure 4, the first world coordinate system can be (X1, Y1, Z1).

In step S302, the external parameters of the depth camera are calibrated according to the first world coordinate system and the image of the first calibration disk taken by the depth camera to obtain the first mapping relationship between the depth camera coordinate system and the first world coordinate system.

In step S303, the second world coordinate system established according to the second calibration plate is acquired. The second calibration disk is placed on the conveyor belt 110 and is in the code reading field of the code reading camera 130.

In step S304, the external parameters of the code reading camera are calibrated according to the second world coordinate system and the image of the second calibration plate taken by the code reading camera to obtain a second mapping between the code reading camera coordinate system and the second world coordinate system relation. As shown in Figure 4, the second world coordinate system can be (X2, Y2, Z2).

In step S305, a third mapping relationship between the first world coordinate system and the second world coordinate system is determined.

In step S306, according to the internal parameters of the code reading camera, a fourth mapping relationship between the coordinate system of the code reading camera and the image coordinate system of the code reading camera is determined.

For example, Figure 4 shows a schematic diagram of the coordinate system in the logistics system. Figure 4 shows the first world coordinate system R1 (X1, Y1, Z1), the second world coordinate system R2 (X2, Y2, Z2), the depth camera coordinate system R3 (X3, Y3, Z3), the code reading camera coordinate It is R4 (X4, Y4, Z4) and the image coordinate system R5 (X5, Y5) of the barcode reading camera 130. Among them, the image coordinate system R5 corresponds to the imaging plane of the code reading camera 130. There is an offset d in the conveying direction of the conveyor belt 110 between the first world coordinate system R1 and the second world coordinate system R2.

For example, the external parameters of the barcode reading camera 130 can be represented by T _CB , that is, the second mapping relationship. For example, it can be expressed as the following matrix form:

Among them, R is, for example, a 3*3 rotation matrix, which represents the rotation transformation parameter between the second world coordinate system and the barcode reading camera coordinate system. T is, for example, a 3*1 translation matrix, which represents the translation transformation parameter between the second world coordinate system and the barcode reading camera coordinate system. I is an orthogonal matrix.

The calibration method of the external parameters of the depth camera 120 is similar to the calibration method of the external parameters of the code reading camera 130. Correspondingly, the first mapping relationship can refer to the related introduction of the second mapping relationship.

The transformation matrix corresponding to the third mapping relationship may be [0,d,0,0] ^T , where d represents the offset of the second world coordinate system relative to the first world coordinate system in the conveying direction of the conveyor belt.

The internal parameters of the code reading camera 130 can be represented by K _C , that is, the fourth mapping relationship. Specifically, it can be expressed as the following matrix form:

Among them, f _x and f _y are the focal length parameters of the code reading camera 130 respectively, and c _x and c _y are the offsets of the camera coordinate system relative to the image coordinate system.

The above steps 301 to S306 are the calibration of each coordinate system and the determination of the mapping relationship. Before the package is inspected, each coordinate system can be calibrated based on step 301-step S306, and the mapping relationship can be determined. Subsequently, when the package needs to be inspected, step S307-step S315 can be executed. In other words, the above steps 301 to S306 only need to be performed once, and it is not necessary to perform the above steps 301 to S306 every time the package is inspected.

In step S307, when the package on the conveyor belt passes through the detection area, the contour of the designated area on the package is recognized, and the detection position of the contour in the detection area is recognized. The designated area includes the barcode of the package, for example, the upper surface of the package or the surface area of the package.

In some embodiments, the designated area is the upper surface of the package, and step S307 may be implemented as method 500.

As shown in FIG. 5, in step S501, when the package passes through the inspection area on the conveyor belt, a depth image of the package is acquired. For example, the depth camera 120 can take pictures of the packages transferred on the conveyor belt 110, and then can obtain the depth images of the packages passing through the inspection area.

In step S502, according to the depth image, the contour of the upper surface of the package and the detection position of the upper surface in the first world coordinate system are determined. Among them, the upper surface of the package can be used as a designated area.

In summary, the method 500 can determine the contour of the upper surface of the package according to the depth image taken by the depth camera 120, so that the upper surface is taken as the designated area, and the detection position of the contour in the detection area is determined.

In some embodiments, the designated area is a single-sided area on the package. Step S307 may be implemented as method 600.

As shown in FIG. 6, in step S601, when the package passes through the inspection area on the conveyor belt, a grayscale image and a depth image of the package are acquired.

In step S602, the contour of the single-surface area wrapped in the grayscale image is determined.

In step S603, according to the contour of the single-surface area in the gray image, the first depth area corresponding to the single-surface area in the depth image is determined.

In step S604, the detection position of the contour of the single area in the first world coordinate system is determined according to the first depth area.

In summary, the method 600 can use the grayscale image to determine the surface area of the package, and then determine the contour and detection position of the surface area based on the grayscale image and the depth image.

In some embodiments, step S502 may be implemented as method 700.

As shown in Fig. 7, in step S701, a three-dimensional model of the package is determined according to the depth image.

In step S702, according to the coordinates of the three-dimensional model in the depth camera coordinate system and the first mapping relationship, the contour of the upper surface of the package and the detection position of the upper surface in the first world coordinate system are determined. Here, the detection position is represented by the coordinates of at least three vertices of the upper surface of the package in the first world coordinate system.

In an optional embodiment, after determining the coordinates of the 3D model of the package in the depth camera coordinate system, the coordinates of the upper surface of the package in the depth camera coordinate system can be obtained according to the coordinates of the 3D model in the depth camera coordinate system. . Based on the coordinates of the upper surface of the package in the depth camera coordinate system, the contour of the upper surface of the package can be determined. In addition, after obtaining the coordinates of the upper surface of the package in the depth camera coordinate system, the coordinates of at least three vertices of the upper surface of the package in the depth camera coordinate system can be selected, and combined with the first mapping relationship, the upper surface of the package can be obtained. The coordinates of at least three vertices in the first world coordinate system.

In summary, the method 700 can determine the three-dimensional model of the package according to the depth image, and then use the three-dimensional model to determine the contour of the designated area (ie, the upper surface) and the detection position of the contour in the first world coordinate system.

In some embodiments, step S502 may be implemented as method 800.

As shown in FIG. 8, in step S801, a gray image corresponding to the depth image is acquired.

In step S802, the contour of the upper surface of the package is determined in the grayscale image. The contour of the upper surface of the package is, for example, a rectangular area.

In step S803, according to the contour of the upper surface of the package in the grayscale image, a second depth region corresponding to the upper surface in the depth image is determined, and at least three vertices of the second depth region are obtained.

In step S804, the coordinates of at least three vertices of the second depth region in the depth camera coordinate system are determined.

In step S805, the detection position of the upper surface of the package in the first world coordinate system is determined according to the coordinates of the at least three vertices of the second depth region in the depth camera coordinate system and the first mapping relationship. Here, the detection position is represented by the coordinates of at least three vertices of the second depth region in the first world coordinate system.

In summary, the method 800 can determine the vertex coordinates of the upper surface of the package according to the grayscale image and the depth image, and then determine the detection position of the upper surface of the package in the first world coordinate system.

When the package on the conveyor belt 110 passes through the inspection area, the method 300 may perform step S308 in addition to step S307. In step S308, the target attribute of the package on the conveyor belt is detected. The target attribute may include, for example, at least one of the volume of the package, the size of the package, the quality of the package, and the face sheet of the package.

In some embodiments, the depth camera 120 may include a line structured light camera. When the depth camera 120 completes the scanning of a package by the online structured light camera, step S308 may obtain a depth image of the package according to the scanned image frame sequence, and determine the size or volume of the package according to the depth image. For example, based on the depth image, step S308 may determine the three-dimensional model of the package, and determine the size or volume of the package according to the three-dimensional model.

In some embodiments, a weighing instrument may also be deployed in the detection area. The weighing instrument can detect the quality of the package. In some embodiments, step S308 may use the grayscale image taken by the depth camera 120 to determine the face sheet of the package.

In step S309, according to the detection position of the contour in the detection area and the conveying speed of the conveyor belt, the code reading position of the contour in the imaging area of the image coordinate system of the code reading camera and the first image identification corresponding to the code reading position are predicted. The barcode reading camera is located downstream of the detection area. The first image identifier is used to identify the package image taken by the code reading camera when the outline reaches the code reading position. The first image identifier is, for example, a frame number or a time stamp. In some embodiments, the first image identifier is, for example, a frame number or a time stamp. The reading position of the contour refers to the coordinate position of the contour in the image coordinate system when at least a part of the contour is in the imaging area. In some embodiments, step S309 may use the detected position as a starting point, and determine at least one offset position according to the conveyor belt speed. Here, when at least part of the designated area of the package enters the field of view of the code reading camera, at least a part of the projection area of the designated area on the package in the image coordinate system is in the imaging area, that is, at least a part of the outline of the designated area is in the imaging area. The position of the projection area is the offset position. The offset position is represented by, for example, the coordinates of the projection points of at least three vertices of the designated area in the image coordinate system.

In some embodiments, step S309 may be implemented as method 900.

As shown in Fig. 9, in step S901, when the detection position is identified, the second image identifier of the image frame currently collected by the code reading camera is determined. The second image is identified as a frame number or a time stamp. The second image identification is generated by the code reading camera 130, for example. For another example, the second image identifier is a frame number or a time stamp added to the image frame currently received from the code reading camera 130 when the computing device 140 determines the detection position.

In step S902, the movement distance of the package in a single collection period of the code reading camera is acquired. For example, the single acquisition period of the code reading camera 130 is T ₁ . The transmission speed is v. The moving distance s=v*T ₁ .

In step S903, the offset position of the contour in the first world coordinate system is determined according to the parameters of the code reading camera, the detection position of the contour in the detection area, and the conveying speed of the conveyor belt. The offset position satisfies: when the contour is at the offset position, the projection position of the contour in the image coordinate system of the code reading camera is in the imaging area. For example, when at least a part of the projection area of the contour at the offset position is in the imaging area in the image coordinate system, step S903 determines that the projection position is in the imaging area. In some embodiments, the resolution of the code reading camera 130 is x*y. The coordinates of the four vertices of the imaging area in the image coordinate system are (0, 0), (x, 0), (0, y), (x, y). When the projection point of at least one vertex of the contour is in the imaging area in the image coordinate system, step S903 may determine that the projection position is in the imaging area.

In some embodiments, step S903 uses the detection position as the starting point and the movement distance as the offset unit to determine the offset position of the package when it is in the code reading field of view. The difference between the offset position and the detection position is equal to an integer number of offset units. For example, the distance between the offset position and the detection position is equal to the sum of N offset units. N is a positive integer.

In some embodiments, step S903 takes the detection position as a starting point, and uses the moving distance as the offset unit, and uses the target position that meets the target condition as the offset position. The target condition is: the difference between the target position and the detection position is equal to an integer number of offset units, and the target position is in the image coordinate system of the code reading camera. (Images) overlap. It should be noted that, during the process of the package from entering the field of view of the code reading camera 130 to leaving the field of view of the code reading camera 130, each image frame captured by the code reading camera 130 includes at least a part of the package. On the surface, in step S903, the package shooting positions corresponding to a part of the image frames or the package shooting positions corresponding to all image frames can be selected as the offset position. Therefore, step S904 can predict one or more reading positions.

In some embodiments, step S903 is based on the coordinates of the offset position in the first world coordinate system (that is, the coordinates of the 4 vertices on the upper surface when the package is placed at the offset position), and according to the second mapping relationship and the third mapping The relationship and the fourth mapping relationship determine the image coordinates of the projection position of the offset position in the image coordinate system of the code reading camera 130.

Taking a vertex of the package as an example, the image coordinates of the projection point of a vertex in the image coordinate system can be calculated according to the following formula:

[u _l ,v _l ,w _l ] ^T = K _C [I|0]T _CB (P _L -[0,d,0,0] ^T +N*[0,Δ _d ,0,0])

Among them, [u _k ,v _k ,w _k ] ^T is the image coordinate of the projection point of a vertex L. P _L represents the coordinates of the vertex L in the first world coordinate system when the package is in the detection position. [0,d,0,0] ^T represents the transformation matrix corresponding to the third mapping relationship. The offset of the second world coordinate system relative to the first world coordinate system in the conveying direction of the conveyor belt is d. P _L -[0,d,0,0] ^T represents the coordinates of the vertex L in the second world coordinate system when the package is at the detection position. (P _L -[0,d,0,0] ^T +N*[0,Δ _d ,0,0]) represents the coordinates of the vertex L in the second world coordinate system when the package is at the offset position. The moving distance Δ _d within a single acquisition cycle 130 of reading the camera. N represents the number of offset units included in the difference between the offset position and the detection position, that is, the ratio of the difference to the offset unit.

T _CB is an external parameter matrix, which represents the external parameters of the code reading camera 130, and can represent the second mapping relationship. For example, the external parameters can be expressed as the following matrix form:

K _C is the internal parameter matrix of the code reading camera 130, which is used to represent the internal parameters of the code reading camera 130, and can represent the fourth mapping relationship.

In step S905, the difference between the offset position and the detection position is calculated, and the number of image frames taken by the code reading camera before the package reaches the code reading position from the detection position is determined according to the number of movement distances contained in the difference. The number of moving distances is consistent with the number of image frames.

In step S906, according to the second image identifier and the number of image frames, the first image identifier corresponding to the code reading position is determined. For example, the second image is identified as frame number I ₂ , and the number of offset units included in the gap between the offset position and the detection position is k ₁ . _{The frame number I 1} =I ₂ +k ₁ identified by the first image. For another example, the second image is identified as a timestamp t ₂ , and the number of offset units included in the difference between the offset position and the detection position is k ₁ . _{The timestamp t 1 of the} first image identification is t 1 =t ₂ +k ₁ *T ₁ , and T ₁ is the time difference between adjacent frames of the code reading camera (ie, the collection period of the code reading camera).

In summary, the method 900 can determine the projection area of the offset position in the package image according to the second mapping relationship, the third mapping relationship, and the fourth mapping relationship, so as to accurately predict that the designated area of the package is at least one code reading field of the code reading field. The position and the identification of the image frame corresponding to each reading position (ie, the first image identification).

In order to illustrate the process of determining the reading position more vividly, an example will be described below with reference to FIGS. 10A-10E.

In the conveyor belt of FIG. 10A, packages B1, B2 and B3 that do not enter the field of view of the depth camera 120 are placed. Among them, the packages B1 and B3 are placed side by side. B2 is behind packages B1 and B3.

FIG. 10B shows the detection position of the package B3 in the detection area determined by the computing device 140. Here, the detection position of the package B3 is represented by the coordinates of the four vertices e1-e4 on the upper surface of the package B3. In the scene where the depth camera 120 is a line structured light camera, the detection position of the package B3 is the position where the package B3 has just left the field of view V1. The computing device 140 may determine the coordinates of the vertices e1-e4 in the first world coordinate system according to the first mapping relationship.

FIG. 10C shows a schematic diagram of a target location of the package B3 predicted by the computing device 140. FIG. 10C only shows the positions of the four vertices e1-e4 on the upper surface of the package B3, and uses the positions of the four vertices to indicate the target position of the package B3. FIG. 10D shows a schematic diagram of the projection of 4 vertices e1-e4 to the image coordinate system (imaging plane). Fig. 10E shows the projection area in the image coordinate system when the package is at the target position in Fig. 10C. The projection area B3' represents the projection area of the upper surface of the package in the image coordinate system when the package is at the target position of Fig. 10C. V2' represents the imaging area, that is, the range of the image generated by the code reading camera 130 in the image coordinate system. It can be seen from FIG. 10E that when the package B3 is at the target position, the projection area of the package B3 in the image coordinate system (that is, the projection area of the designated area in the image coordinate system) is in the imaging area. In addition, the computing device 140 may determine that the difference between the target position of FIG. 10C and the detection position of the package B3 includes an integer number of offset units. Therefore, the computing device 140 may use the target position shown in FIG. 10C as an offset position.

In some embodiments, the method 300 may further include step S310.

In step S310, when the package image corresponding to the first image identifier is acquired, barcode recognition is performed on the package image.

When the barcode in the package image is identified in step S310, the method 300 may execute step S311 to position the identified barcode with the barcode reading position.

In some embodiments, step S311 may be implemented as method 1100.

As shown in FIG. 11, in step S1101, the barcode area of the barcode in the package image is determined. For example, step S1101 may determine the image coordinates of each barcode area in the package image. For example, FIG. 12A shows a schematic diagram of a package image according to some embodiments of the present application. The package image P1 in FIG. 12A includes barcode areas C1 and C2.

In step S1102, it is determined whether the barcode area belongs to the area corresponding to the barcode reading position.

In some embodiments, based on the coordinates of the offset position of the outline of the designated area on the package in the first world coordinate system, and according to the second mapping relationship, the third mapping relationship, and the fourth mapping relationship, step S1102 may determine that the package is in When shifting the position, the projection area of the designated area in the image coordinate system (ie the area corresponding to the reading position). Here, in step S1102, based on the third mapping relationship (that is, the mapping relationship between the first world coordinate system and the second world coordinate system), the coordinates of the offset position in the second world coordinate system can be determined. Based on the second mapping relationship (ie, the mapping relationship between the second world coordinate system and the camera coordinate system of the code reading camera), step S1102 can determine the coordinates of the offset position in the second world coordinate system in the code reading camera coordinate system. Based on the fourth mapping relationship (ie the mapping relationship between the code reading camera coordinate system and the image coordinate system of the code reading camera) and the coordinates of the offset position in the code reading camera coordinate system, step S1102 determines the projection of the designated area of the package at the offset position The coordinates of the projection area to the image coordinate system (ie, the reading position).

For example, for package B3, step S1102 can determine the projection area of package B3 in the package image (ie, the area corresponding to the barcode reading position). Figure 12B shows the projection area B3" of the package B3. In step S1102, it can be determined that the barcode area of the barcode C1 is outside the projection area B3", and the barcode area of the barcode C2 belongs to the projection area B3".

When it is determined in step S1102 that at least a part of the barcode area belongs to the area corresponding to the barcode reading position, that is, when it is determined that the barcode area belongs to the area corresponding to the barcode reading position, the method 1100 may execute step S1103 to determine the position matching between the barcode and the barcode reading position . Taking Figure 12B as an example, it can be determined that the barcode C1 and the parcel B3 have a position matching between the barcode reading positions.

When it is determined in step S1102 that the barcode region is outside the region corresponding to the barcode reading position, that is, when it is determined that the barcode region does not belong to the region corresponding to the barcode reading position, the method 1100 may execute step S1104 to determine that the barcode and the barcode reading position do not match. .

In summary, the method 1100 can determine whether the barcode and the barcode reading position match according to the position relationship between the barcode reading position and the barcode. Here, the barcode matching with the reading position can be understood as: the barcode is on the package corresponding to the reading position. The mismatch between the barcode and the barcode reading position can be understood as: the barcode does not belong to the package corresponding to the barcode reading position.

In some embodiments, the method 300 may further include steps S312 and S313. In step S312, when the barcode reading position matches the barcode, the matched barcode is associated with the target attribute. Taking FIG. 12B as an example, in step S312, the barcode C2 and the target attribute of the package B3 may be associated and stored.

In step S313, the matched barcode is associated with the package.

In summary, according to the package detection method 300 of the present application, when the detection position of the package is detected, the code reading position of the package in the imaging area and the first image identification corresponding to the code reading position can be predicted. On this basis, according to the package detection method 300 of the present application, by matching the barcode and the barcode reading position in the package image, the association relationship between the package and the barcode can be determined, and the target attribute of the package can be associated with the barcode. Yes, after determining the attributes of the package, you can also associate the attributes of the package with the barcode.

In some embodiments, the method 300 may further include step S314 of obtaining a panoramic image of the extended processing area of the conveyor belt. Among them, the extended processing area is located downstream of the barcode reading field.

In step S315, according to the detected position and the transmission speed, continuously update the predicted position of the package in the extended processing area over time, and add a tracking frame to the panoramic image according to the predicted position. Wherein, when the target attribute of the package is associated with the barcode, the tracking box can be presented as the first color. When the target attribute of the package is not associated with the barcode, step S315 may present the tracking box in the second color. For example, the first color is green and the second color is red.

Figure 13A shows a schematic diagram of a logistics system according to some embodiments of the present application. FIG. 13A further adds a camera 150 on the basis of FIG. 1. The camera 150 may output a sequence of image frames to the computing device 140, that is, a sequence of panoramic image frames. The camera 150 is downstream of the code reading camera 130. The field of view of the camera 150 is V3. The field of view V3 can cover the extended processing area S3 of the conveyor belt. The computing device 140 can update the predicted position of each package in the extended processing area according to the detected position and the transmission speed, and add a tracking frame to the panoramic image. In this way, the computing device 140 can track the location of the package through the tracking box. In addition, the computing device 140 can display different states of the package by displaying the tracking frame of the package in different colors. In this way, the staff can easily determine that the target attribute of the package corresponding to the tracking frame of the second color is not associated with a barcode, that is, there is no identifiable barcode on the upper surface of the package corresponding to the tracking frame of the second color. For example, the case where there is no identifiable bar code on the upper surface of the package is, for example, the bar code on the upper surface of the package is incomplete, the package does not have a bar code, or the package bar code is on the side or bottom of the package. The staff can perform code supplementation and other processing for packages that do not have identifiable barcodes. For example, FIG. 13B shows a schematic diagram of a panoramic image according to some embodiments of the present application. In FIG. 13B, the tracking frame M1 may be presented in green, for example, and the tracking frame M2 may be presented in red, for example. According to the color of the tracking box, the staff can quickly find the red package and perform operations such as complementing the code.

In summary, according to the package detection method 300 of the present application, it is possible to track the packages leaving the code reading field through steps S314 and S315, and by presenting different colors to prompt the package status, it can greatly improve the detection of abnormal packages ( If the target attribute is not associated with the barcode package), it is convenient to handle.

Figure 14 shows a schematic diagram of a logistics system according to some embodiments of the present application. Figure 14 shows an array of code reading cameras. The array includes, for example,

code reading cameras

130, 160, and 170. In the array, the field of view of adjacent code reading cameras can be adjacent or partially overlapped. The computing device 140 can predict the barcode reading position of a package in the

barcode reading cameras

130, 160, and 170, and associate the target attribute of the package with the barcode according to the package image of each barcode reading camera. In this way, multiple barcode reading cameras are used to associate target attributes with barcodes. The computing device 140 can compare the association results (ie, the association results between the target attribute and the barcode) corresponding to different barcode reading cameras, so as to improve the accuracy of associating the target attribute with the barcode. For example, the association result corresponding to the

barcode reading camera

130 and 160 is the same, and the association result corresponding to the barcode reading camera 170 is different from the association result corresponding to the barcode reading camera 130, then the computing device 140 uses the association result corresponding to the barcode camera 130 (160) The result shall prevail.

FIG. 15 shows a schematic diagram of an apparatus 1500 for detecting packages according to some embodiments of the present application. Here, the apparatus 1500 may be deployed in the computing device 140, for example.

As shown in FIG. 15, the device 1500 for detecting packages may include: a detection unit 1501, a prediction unit 1502, a barcode recognition unit 1503, a matching unit 1504, and an association unit 1505.

The detection unit 1501 is used to identify the contour of the designated area on the package when the package passes through the detection area on the conveyor belt, and to identify the detection position of the contour in the detection area. The designated area includes the barcode of the package.

The prediction unit 1502 is used for predicting the reading position of the contour in the imaging area of the image coordinate system of the code reading camera and the first image identifier corresponding to the reading position according to the detection position of the contour in the detection area and the conveying speed of the conveyor belt. The code reading camera is located downstream of the detection area, and the first image identifier is used to identify the package image taken by the code reading camera when the contour reaches the code reading position.

The barcode recognition unit 1503 is configured to perform barcode recognition on the package image when the package image corresponding to the first image identifier is acquired.

When the barcode recognition unit 1503 recognizes the barcode in the package image, the matching unit 1504 is used to match the barcode reading position with the recognized barcode.

When the matching unit 1504 determines that the barcode reading position matches the barcode, the associating unit 1505 is used to associate the matched barcode with the package.

In summary, the device 1500 for detecting packages according to the present application can predict the code reading position of the designated area of the package in the imaging area and the first image identification corresponding to the code reading position when the detection position of the package is detected. Here, the barcode reading position can be considered as the projection position of the designated area of the package in the image coordinate system when the package image corresponding to the first image identifier is acquired. On this basis, the device 1500 for detecting a package according to the present application can determine the relationship between the package and the barcode by matching the barcode and the barcode reading position in the package image. Correspondingly, after determining the attributes of the package, it can also Associate the attributes of the package with the barcode.

FIG. 16 shows a schematic diagram of an apparatus 1600 for detecting packages according to some embodiments of the present application. Here, the apparatus 1600 may be deployed in the computing device 140, for example.

As shown in FIG. 16, the device 1600 for detecting packages may include: a detection unit 1601, a prediction unit 1602, a barcode recognition unit 1603, a matching unit 1604, an association unit 1605, a first calibration unit 1606, and a second calibration unit 1607.

The first calibration unit 1606 can obtain the first world coordinate system established according to the first calibration disk. The first calibration disc is placed on the conveyor belt and is in the field of view of the depth camera. According to the first world coordinate system and the image of the first calibration disk taken by the depth camera, the first calibration unit 1606 can calibrate the external parameters of the depth camera to obtain the first mapping relationship between the depth camera coordinate system and the first world coordinate system.

The second calibration unit 1607 acquires the second world coordinate system established according to the second calibration disk. The second calibration disc is placed on the conveyor belt and is in the reading field of the code reading camera. According to the second world coordinate system and the image of the second calibration plate taken by the code-reading camera, the second calibration unit 1607 can calibrate the external parameters of the code-reading camera to obtain the second between the code-reading camera coordinate system and the second world coordinate system. Mapping relations.

In addition, the second calibration unit 1607 can also determine the third mapping relationship between the first world coordinate system and the second world coordinate system. According to the internal parameters of the code reading camera, the second calibration unit 1607 can determine the fourth mapping relationship between the code reading camera's coordinate system and the image coordinate system of the code reading camera.

The detection unit 1601 is used to identify the contour of the designated area on the package and the detection position of the contour in the detection area when the package passes through the detection area on the conveyor belt. The designated area includes the barcode of the package.

In some embodiments, the detection unit 1601 may obtain a depth image of the package when the package passes through the detection area on the conveyor belt. According to the depth image, the detection unit 1601 determines the contour of the upper surface of the package and the detection position of the upper surface in the first world coordinate system. Wherein, the upper surface is the designated area.

In some embodiments, the detection unit 1601 may determine a three-dimensional model of the package according to the depth image. According to the coordinates of the three-dimensional model in the depth camera coordinate system and the first mapping relationship, the detection unit 1601 can determine the contour of the upper surface of the package and the detection position of the upper surface in the first world coordinate system. The detection position is represented by the coordinates of at least three vertices of the upper surface of the package in the first world coordinate system.

In some embodiments, the detection unit 1601 may obtain a grayscale image corresponding to the depth image. The detection unit 1601 can determine the contour of the upper surface of the package in the gray image. According to the contour of the upper surface of the package in the grayscale image, the detection unit 1601 can determine the second depth region corresponding to the upper surface in the depth image, and obtain at least three vertices of the second depth region. The detection unit 1601 may determine the coordinates of at least three vertices of the second depth region in the depth camera coordinate system. According to the coordinates of the at least three vertices of the second depth region in the depth camera coordinate system and the first mapping relationship, the detection unit 1601 can determine the detection position of the upper surface of the package in the first world coordinate system. The detection position is represented by the coordinates of at least three vertices of the second depth region in the first world coordinate system.

In some embodiments, the detection unit 1601 may obtain a grayscale image and a depth image of the package when the package passes through the detection area on the conveyor belt. The detection unit 1601 can determine the contour of the single-surface area wrapped in the grayscale image. According to the contour of the single-surface area wrapped in the grayscale image, the detection unit 1601 can determine the first depth area corresponding to the single-surface area in the depth image. According to the first depth area, the detection unit 1601 can determine the detection position of the contour of the single area in the first world coordinate system. The single area is the designated area.

In some embodiments, the detection unit 1601 may also detect the target attribute of the package when the package passes through the detection area on the conveyor belt. The target attributes include at least one of the volume of the package, the size of the package, the quality of the package, and the face sheet of the package.

The prediction unit 1602 is used for predicting the code reading position of the contour in the imaging area of the image coordinate system of the code reading camera and the first image identifier corresponding to the code reading position according to the detection position of the contour in the detection area and the conveying speed of the conveyor belt. The code reading camera is located downstream of the detection area, and the first image identifier is used to identify the package image taken by the code reading camera when the outline reaches the code reading position.

In some embodiments, the detection position of the contour in the detection area is the coordinate of the contour in the first world coordinate system.

When the detection unit 1601 recognizes the detection position, the prediction unit 1602 can determine the second image identifier of the image frame collected by the barcode reading camera at the current moment. The second image is identified as a frame number or a time stamp. The prediction unit 1602 can obtain the moving distance of the package in a single collection period of the code reading camera. On this basis, the prediction unit 1602 can determine the offset position of the contour in the first world coordinate system according to the parameters of the code reading camera, the detection position of the contour in the detection area, and the conveying speed of the conveyor belt. The offset position satisfies: when the contour is at the offset position, the projection position of the contour in the image coordinate system of the code reading camera is in the imaging area. In this way, the prediction unit 1602 can use the projection position of the contour in the image coordinate system of the code reading camera as the code reading position when the contour is at the offset position.

In addition, the prediction unit 1602 may calculate the difference between the offset position and the detection position, and determine the number of image frames taken by the code reading camera before the package reaches the code reading position from the detection position according to the number of movement distances included in the difference. According to the second image identifier and the number of image frames, the prediction unit 1602 can determine the first image identifier corresponding to the code reading position.

The barcode recognition unit 1603 is configured to perform barcode recognition on the package image when the package image corresponding to the first image identifier is acquired.

When the barcode recognition unit 1603 recognizes the barcode in the package image, the matching unit 1604 is used to match the barcode reading position with the barcode.

In some embodiments, the matching unit 1604 may determine the barcode area of the barcode in the package image. The matching unit 1604 determines whether the barcode area belongs to the area corresponding to the barcode reading position. When at least a part of the barcode area belongs to the area corresponding to the barcode reading position, the matching unit 1604 determines the position matching between the barcode and the barcode reading position. When the barcode area is outside the area corresponding to the barcode reading position, the matching unit 1604 determines that the barcode and the barcode reading position do not match.

When the matching unit 1604 determines that the barcode reading position matches the barcode, the associating unit 1605 is used to associate the matched barcode with the package. In addition, when the barcode reading position matches the barcode, the associating unit 1605 can associate the matched barcode with the target attribute. In some embodiments, the apparatus 1600 may further include a tracking unit 1608. The tracking unit 1608 can acquire a panoramic image of the extended processing area of the conveyor belt. Among them, the extended processing area is located downstream of the barcode reading field. Based on the detected position and the transmission speed, the prediction unit 1602 continuously updates the predicted position of the package in the extended processing area over time. The tracking unit 1608 may add a tracking frame to the panoramic image according to the predicted position. Wherein, when the target attribute of the package is associated with the barcode, the tracking unit 1608 presents the tracking frame as the first color. When the target attribute of the package is not associated with the barcode, the tracking unit 1608 presents the tracking frame in the second color.

A more specific implementation manner of the apparatus 1600 is similar to that of the method 300, and will not be repeated here.

Figure 17 shows a schematic diagram of a computing device according to some embodiments of the present application. As shown in FIG. 17, the computing device includes one or more processors (CPU) 1702, a communication module 1704, a memory 1706, a user interface 1710, and a communication bus 1708 for interconnecting these components.

The processor 1702 may receive and send data through the communication module 1704 to implement network communication and/or local communication.

The user interface 1710 includes one or more output devices 1712, which includes one or more speakers and/or one or more visual displays. The user interface 1710 also includes one or more input devices 1714. The user interface 1710 may, for example, receive instructions from a remote controller, but is not limited to this.

The memory 1706 may be a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state storage devices; or a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, or flash memory devices, Or other non-volatile solid-state storage devices.

The memory 1706 stores an instruction set executable by the processor 1702, including:

Operating system 1716, including programs for processing various basic system services and performing hardware-related tasks;

The application 1718 includes various programs for realizing the aforementioned package detection, for example, it may include a package detection device 1500 or 1600. Such a program can implement the processing procedures in the above-mentioned examples, and may include, for example, a method of detecting packages.

In addition, each embodiment of the present application can be implemented by a data processing program executed by a data processing device such as a computer. Obviously, the data processing program constitutes this application. In addition, a data processing program usually stored in a storage medium is executed by directly reading the program out of the storage medium or by installing or copying the program to a storage device (such as a hard disk and/or a memory) of the data processing device. Therefore, such a storage medium also constitutes the present application. The storage medium can use any type of recording method, such as paper storage medium (such as paper tape, etc.), magnetic storage medium (such as floppy disk, hard disk, flash memory, etc.), optical storage medium (such as CD-ROM, etc.), magneto-optical storage medium ( Such as MO, etc.) and so on.

Therefore, this application also discloses a non-volatile storage medium in which a program is stored. The program includes instructions that, when executed by the processor, cause the computing device to execute the method of detecting packages according to the present application.

In addition, the method steps described in this application can be implemented by data processing programs, but also by hardware, for example, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded micro-controllers.器, etc. to achieve. Therefore, this kind of hardware that can implement the method described in this application can also constitute this application.

The above are only preferred embodiments of this application, and are not intended to limit this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in the protection of this application. Within the range.

Claims

A method of detecting packages, including:

When a package on a conveyor belt passes through a detection area, identifying the contour of a designated area on the package, and identifying the detection position of the contour in the detection area, the designated area including the barcode of the package;

According to the detection position of the contour in the detection area and the conveying speed of the conveyor belt, it is predicted that the reading position of the contour in the imaging area of the image coordinate system of the code reading camera and the first corresponding to the reading position An image identification, the code reading camera is located downstream of the detection area, and the first image identification is used to identify a package image taken by the code reading camera when the outline reaches the barcode reading position;

When the package image corresponding to the first image identifier is acquired, perform barcode recognition on the package image;

When the barcode in the package image is recognized, the barcode reading position is matched with the recognized barcode;

When the barcode reading position matches the barcode, the matched barcode is associated with the package.
The method according to claim 1, wherein the detection position of the contour in the detection area is the coordinate of the contour in the first world coordinate system; Detecting the position and the conveying speed of the conveyor belt, predicting the barcode reading position of the contour in the imaging area of the image coordinate system of the barcode reading camera and the first image identifier corresponding to the barcode reading position include:

When the detection position is identified, the second image identifier of the image frame collected by the barcode reading camera at the current moment is determined, and the second image identifier is the frame number or the time stamp;

Obtain the moving distance of the package in a single collection cycle of the barcode reading camera;

Determine the offset position of the contour in the first world coordinate system according to the parameters of the code reading camera, the detection position of the contour in the detection area, and the conveying speed of the conveyor belt. It is satisfied that when the contour is at the offset position, the projection position of the contour in the image coordinate system of the code reading camera is in the imaging area;

When the contour is at the offset position, the projection position of the contour in the image coordinate system of the code reading camera is taken as the code reading position;

Calculate the difference between the offset position and the detection position, and according to the number of the movement distance included in the difference, determine that the code reading camera reaches the reading code when the package is from the detection position The number of image frames taken before the location;

According to the second image identifier and the number of image frames, the first image identifier corresponding to the code reading position is determined.
The method of claim 1, further comprising:

When the package on the conveyor belt passes through the detection area, the target attribute of the package is detected, and the target attribute includes at least one of the volume of the package, the size of the package, the quality of the package, and the face sheet of the package;

When the barcode reading position matches the barcode, the matched barcode is associated with the target attribute.
The method according to claim 1, wherein the position matching the barcode reading position with the barcode includes:

Determining the barcode area of the barcode in the package image;

Determining whether the barcode area belongs to the area corresponding to the barcode reading position;

When at least a part of the barcode area belongs to the area corresponding to the barcode reading position, determining the position match between the barcode and the barcode reading position;

When the barcode area is outside the area corresponding to the barcode reading position, it is determined that the barcode and the barcode reading position do not match.
The method of claim 1, further comprising:

Acquiring a panoramic image of the extended processing area of the conveyor belt, wherein the extended processing area is located downstream of the code reading field in the conveying direction of the conveyor belt;

According to the detection position and the transmission speed, continuously update the predicted position of the package in the extended processing area over time, and add a tracking frame to the panoramic image according to the predicted position;

Wherein, when the package is associated with a barcode, the tracking frame is presented in a first color, and when the package is not associated with a barcode, the tracking frame is presented in a second color.
The method according to claim 1, wherein, when the package on the conveyor belt passes through the detection area, identifying the contour of the designated area on the package and recognizing the detection position of the contour in the detection area comprises:

When the package on the conveyor belt passes through the detection area, the depth image of the package is acquired; according to the depth image, the contour of the upper surface of the package and the detection position of the upper surface in the first world coordinate system are determined, wherein the The upper surface is the designated area; or

When the package on the conveyor belt passes through the detection area, obtain the grayscale image and depth image of the package; determine the contour of the single area of the package in the grayscale image; according to the surface of the package in the grayscale image The contour of a single area determines the first depth area in the depth image corresponding to the single area of the surface; according to the first depth area, the detection position of the contour of the single area of the surface in the first world coordinate system is determined, where The single surface area is the designated area.
The method of claim 6, further comprising:

Acquiring the first world coordinate system established according to the first calibration disk, the first calibration disk being placed on the conveyor belt and in the field of view of the depth camera;

Calibrate the external parameters of the depth camera according to the first world coordinate system and the image of the first calibration disk taken by the depth camera to obtain the first mapping relationship between the depth camera coordinate system and the first world coordinate system;

Acquiring a second world coordinate system established according to a second calibration disk, the second calibration disk being placed on a conveyor belt and in the code reading field of view of the code reading camera;

Calibrate the external parameters of the code reading camera according to the second world coordinate system and the image of the second calibration disk taken by the code reading camera to obtain a second mapping relationship between the code reading camera coordinate system and the second world coordinate system;

Determine the third mapping relationship between the first world coordinate system and the second world coordinate system;

According to the internal parameters of the code-reading camera, the fourth mapping relationship between the coordinate system of the code-reading camera and the image coordinate system of the code-reading camera is determined.
The method according to claim 7, wherein the determining the contour of the upper surface of the package and the detection position of the upper surface in the first world coordinate system according to the depth image comprises:

According to the depth image, determine the three-dimensional model of the package; according to the coordinates of the three-dimensional model in the depth camera coordinate system and the first mapping relationship, determine the contour of the upper surface of the package and the coordinates of the upper surface in the first world The detection position in the system, the detection position is represented by the coordinates of at least three vertices of the upper surface of the package in the first world coordinate system; or

Acquiring a grayscale image corresponding to the depth image;

Determining the contour of the upper surface of the package in the grayscale image;

Determining a second depth area corresponding to the upper surface in the depth image according to the contour of the upper surface of the package in the grayscale image, to obtain at least three vertices of the second depth area;

Determining the coordinates of at least three vertices of the second depth region in the depth camera coordinate system;

According to the coordinates of the at least three vertices of the second depth area in the depth camera coordinate system and the first mapping relationship, the detection position of the upper surface of the package in the first world coordinate system is determined, and the detection position It is represented by the coordinates of at least three vertices of the second depth region in the first world coordinate system.
A device for detecting packages, including:

The detection unit is configured to identify the contour of a designated area on the package when the package passes through the detection area on the conveyor belt, and identify the detection position of the contour in the detection area, and the designated area includes the barcode of the package;

A predicting unit for predicting the barcode reading position and the barcode reading of the contour in the imaging area of the image coordinate system of the barcode reading camera based on the detection position of the contour in the detection area and the conveying speed of the conveyor belt A first image identifier corresponding to a position, the code reading camera is located downstream of the detection area, and the first image identifier is used to identify a package image taken by the code reading camera when the outline reaches the code reading position;

The barcode recognition unit is configured to perform barcode recognition on the package image when the package image corresponding to the first image identifier is obtained;

When the barcode recognition unit recognizes the barcode in the package image, the matching unit is used to match the barcode reading position with the recognized barcode;

When the matching unit determines that the barcode reading position matches the barcode, the associating unit is used to associate the matched barcode with the package.
A computing device including:

Memory

processor;

A program, stored in the memory and configured to be executed by the processor, the program including instructions for executing the method of any one of claims 1-8.
A storage medium storing a program, the program including instructions, which when executed by a computing device, cause the computing device to execute the method according to any one of claims 1-8.
A logistics system including:

The computing device according to claim 10;

Conveyor belt

Depth camera

Code reading camera.