CN112907668B

CN112907668B - Method and device for identifying stacking box bodies in stack and robot

Info

Publication number: CN112907668B
Application number: CN202110218320.4A
Authority: CN
Inventors: 魏海永; 段文杰; 盛文波; 丁有爽; 邵天兰
Original assignee: Mech Mind Robotics Technologies Co Ltd
Current assignee: Mech Mind Robotics Technologies Co Ltd
Priority date: 2021-02-26
Filing date: 2021-02-26
Publication date: 2024-01-30
Anticipated expiration: 2041-02-26
Also published as: CN112907668A

Abstract

The invention discloses a method and a device for identifying a stacking box in a stack and a robot, wherein the method comprises the following steps: extracting the highest layer point cloud of the stack type according to the depth image shot from the upper part of the stack type, and extracting the highest layer edge point cloud according to the highest layer point cloud of the stack type; projecting the highest layer edge point cloud to a two-dimensional plane to obtain an image to be identified of the target; performing edge matching processing according to the standard box image and the target image to be identified, and determining each box image area in the target image to be identified according to an edge matching result; and determining the position information of each group of stacked boxes in the highest layer of the stacked type according to the image areas of each box. By the method, the accuracy of identifying the stacking box bodies in the stacking type can be improved, so that the accuracy of the related operation of the subsequent stacking type is ensured.

Description

Method and device for identifying stacking box bodies in stack and robot

Technical Field

The invention relates to the technical field of warehouse logistics, in particular to a method for identifying a stacking box in a stack, a robot and computing equipment.

Background

The storage logistics intelligentization comprises the operations of disassembling, stacking, integrating and the like of box stack types through a mechanical arm, and the stack types need to be identified before the operations. Because the highest layer of boxes in the stack type are easiest to operate, and stacking suggestions need to be given based on the stacking condition of the highest layer of boxes when stacking the boxes, the edges of the boxes of the highest layer need to be identified before the manipulator operates the stack type, and then the placing condition of the boxes is identified, so that references for disassembling, stacking and integrating execution schemes are given.

The inventors have found in the course of implementing the invention that: in the prior art, the identification of the stack type edge is inaccurate, so that the identification of the box placement condition is also inaccurate, for example, a gap in the middle of a single box is judged to be a gap between two boxes, and then the subsequent operation is wrong.

Disclosure of Invention

The present invention has been made in view of the above-mentioned problems, and has as its object to provide a method, an apparatus and a robot for identifying a stack box in a stack, which overcome or at least partially solve the above-mentioned problems.

According to one aspect of the present invention, there is provided a method for identifying a stack box in a stack, comprising:

extracting the highest layer point cloud of the stack type according to the depth image shot from the upper part of the stack type, and extracting the highest layer edge point cloud according to the highest layer point cloud of the stack type;

projecting the highest layer edge point cloud to a two-dimensional plane to obtain an image to be identified of the target;

performing edge matching processing according to the standard box image and the target image to be identified, and determining each box image area in the target image to be identified according to an edge matching result;

and determining the position information of each group of stacked boxes in the highest layer of the stacked type according to the image areas of each box.

Optionally, performing edge matching processing according to the standard box image and the target image to be identified, and determining each box image area in the target image to be identified according to the edge matching result further includes:

step S0, let i=1;

step S1, carrying out edge matching processing according to a standard box image and an ith wheel of to-be-identified image, and determining the image area of each ith wheel of the edge box according to an edge matching result;

when the value of i is 1, the ith round of image to be identified is a target image to be identified;

s2, deleting each ith round of edge box body image area from the ith round of images to be identified, and judging whether the matching termination condition is met or not according to the deleted residual images; if yes, confirming that the identification of the box body image area is finished; if not, i is assigned as i+1, the rest images are determined as i-th round images to be identified, and the step S1 is executed in a jumping mode.

Optionally, performing edge matching processing according to the standard box image and the ith round of image to be identified, and determining each ith round of edge box image area according to the edge matching result further includes:

moving and traversing in the ith round of images to be identified by using the standard box body image;

when the standard box image moves to any traversing position angle, calculating the edge coincidence degree of the standard box image and the ith wheel of image to be identified;

And determining the image areas of the ith-wheel edge box body in the ith-wheel image to be identified according to the edge overlapping ratio.

Optionally, calculating the edge overlap ratio of the standard box image and the i-th round of image to be identified further includes:

and calculating the edge contact ratio of any group of adjacent edges of the standard box image and any group of adjacent edges of the ith round of image to be identified.

Optionally, performing the moving traversal in the ith round of the image to be identified using the standard box image further comprises:

moving and traversing in each preset area of the ith round of images to be identified by using the standard box images;

according to the edge overlap ratio, determining each ith-wheel edge box image area in the ith-wheel image to be identified further comprises:

and aiming at any preset area, determining an ith wheel edge box image area in the preset area according to the position angle information of the standard box image when the edge overlap ratio is highest.

Optionally, each preset area is obtained by dividing according to preset origin position information, preset traversal step length information and preset traversal angle information.

Optionally, determining each ith-wheel edge box image area in the ith-wheel image to be identified according to the edge overlap ratio further includes:

Calculating the length proportion between each edge in the ith round of images to be identified and the corresponding edge of the standard box body image;

calculating the coincidence credibility of the standard box image when the standard box image moves to the position angle according to the edge coincidence degree and the length proportion between each edge in the ith wheel of image to be identified and the corresponding edge of the standard box image;

and determining the image areas of the ith-wheel edge box body in the ith-wheel images to be identified according to the coincidence reliability.

Optionally, the method further comprises:

and comparing each 1 st-wheel edge box body image area with the highest-layer point cloud, and filtering at least one 1 st-wheel edge box body image area if the area of the at least one 1 st-wheel edge box body image area exceeding the highest-layer point cloud reaches a preset proportion.

Optionally, the method further comprises:

acquiring a color image shot from the upper part of the stack, and performing edge extraction processing on the color image to obtain a color edge image;

synthesizing the color edge image and the target image to be identified to obtain a highest-layer edge synthesized image;

then performing edge matching processing according to the standard box image and the target image to be identified further comprises:

and performing edge matching processing according to the standard box body image and the highest layer edge synthetic image.

Optionally, after determining each ith round of edge box image area according to the edge matching result, the method further comprises:

according to any ith wheel edge box image area, unstacking configuration information of a target box corresponding to the ith wheel edge box image area is determined, so that unstacking operation is carried out on the target box according to the unstacking configuration information;

wherein, the unstacking configuration information includes: and capturing position information and moving direction information.

Optionally, according to any ith wheel edge box image area, determining unstacking configuration information of the target box corresponding to the ith wheel edge box image area further includes:

and determining the moving direction information and the grabbing position information of the target box according to at least one group of adjacent edges, which are overlapped with the positions of at least one group of adjacent edges of the image to be identified of the ith wheel, in the image area of the ith wheel edge box.

Optionally, projecting the highest layer edge point cloud onto the two-dimensional plane further comprises:

and projecting the edge point cloud of the highest layer to a two-dimensional plane according to the height information of the highest layer and the internal reference information of the camera.

According to another aspect of the present invention, there is provided an identification device for a stack box in a stack type, including:

The edge extraction module is suitable for extracting the highest layer point cloud of the stack type according to the depth image shot from the upper side of the stack type and extracting the highest layer edge point cloud according to the highest layer point cloud of the stack type;

the conversion module is suitable for projecting the highest-layer edge point cloud to a two-dimensional plane to obtain a target image to be identified;

the edge matching module is suitable for carrying out edge matching processing according to the standard box image and the target image to be identified, and determining each box image area in the target image to be identified according to an edge matching result;

and the positioning module is suitable for determining the position information of each stacking box body in the highest stacking layer according to the image areas of each box body.

Optionally, the apparatus further comprises:

the correction module is suitable for judging whether the shape of the target image to be identified is rectangular or not;

if not, correcting the target to-be-identified image to correct the shape of the target to-be-identified image into a rectangle, and obtaining a corrected target to-be-identified image;

the edge matching module is further adapted to:

and carrying out edge matching processing according to the standard box body image and the corrected target image to be identified.

Optionally, the edge matching module is further adapted to perform the steps of:

Step S0, let i=1;

Optionally, the edge matching module is further adapted to:

Optionally, the apparatus further comprises:

and the filtering module is suitable for comparing each 1 st round of edge box body image area with the highest layer point cloud, and filtering at least one 1 st round of edge box body image area if the area of the at least one 1 st round of edge box body image area exceeding the highest layer point cloud reaches a preset proportion.

Optionally, the edge extraction module is further adapted to:

the edge matching module is further adapted to: and performing edge matching processing according to the standard box body image and the highest layer edge synthetic image.

Optionally, the apparatus further comprises:

the unstacking configuration module is suitable for determining unstacking configuration information of a target box corresponding to any ith wheel edge box image area according to any ith wheel edge box image area so as to perform unstacking operation on the target box according to the unstacking configuration information; wherein, the unstacking configuration information includes: and capturing position information and moving direction information.

Optionally, the unstacking configuration module is further adapted to:

Optionally, the conversion module is further adapted to:

According to another aspect of the invention, a robotic system is provided comprising the identification means of the stacking boxes in a stack as described above.

According to yet another aspect of the present invention, there is provided a computing device comprising: the device comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete communication with each other through the communication bus;

the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the identification method of the stacking box body in the stacking.

According to still another aspect of the present invention, there is provided a computer storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the method for identifying a stacking box in a stack type as described above.

The invention relates to a method and a device for identifying a stacking box in a stack and a robot, wherein the method comprises the following steps: extracting the highest layer point cloud of the stack type according to the depth image shot from the upper part of the stack type, and extracting the highest layer edge point cloud according to the highest layer point cloud of the stack type; projecting the highest layer edge point cloud to a two-dimensional plane to obtain an image to be identified of the target; performing edge matching processing according to the standard box image and the target image to be identified, and determining each box image area in the target image to be identified according to an edge matching result; and determining the position information of each group of stacked boxes in the highest layer of the stacked type according to the image areas of each box. By the method, the accuracy of identifying the stacking box bodies in the stacking type can be improved, so that the accuracy of the related operation of the subsequent stacking type is ensured.

The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:

FIG. 1 shows a flowchart of a method for identifying a stacking box in a stack according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating a method for identifying a stacking box in a stack according to another embodiment of the present invention;

FIG. 3a shows a schematic stacking view of the various bins of the uppermost tier of the stack;

FIG. 3b shows a schematic diagram of an edge matching process of an image to be identified of a target;

FIG. 4 shows a schematic diagram of the positional relationship of a standard box image and a target image to be identified;

Fig. 5 is a schematic structural view of an identification device for stacking boxes in a stack according to another embodiment of the present invention;

FIG. 6 illustrates a schematic diagram of a computing device provided by an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

Fig. 1 shows a flowchart of a method for identifying stacking boxes in a stack type according to an embodiment of the present invention, where the method of the present embodiment is applicable to a stack type with the same specification for each stacking box, as shown in fig. 1, and includes the following steps:

step S110, extracting the highest layer point cloud of the stack type according to the depth image shot from the upper part of the stack type, and extracting the highest layer edge point cloud according to the highest layer point cloud of the stack type.

The image collector (such as a camera) is located above the stack and is used for shooting the depth image, and the camera can shoot directly above the stack or shoot at any angle not directly above the stack.

The depth image refers to an image in which depth values from an image collector to points in a scene are used as pixel values, and the depth values are used for reflecting the definition of how far or how far an object is from the image collector. Since the camera is not necessarily photographed from directly above the stack, but may be photographed from obliquely above the stack, the depth value may be a value reflecting not only the distance of the object from the camera of the camera, but also the distance in the gravitational direction, and the depth value may be a value reflecting the distance between the object and the camera in the vertical direction of the horizontal plane of the object lifting device (e.g., basket, conveyor, tray, etc.), and in practice, the conveyor may be inclined, and the tray may be inclined. Therefore, the depth direction does not necessarily coincide with the camera shooting direction, but is based on a vertical line of the object horizontal plane. If the camera is directly above, the depth direction is the depth direction consistent with the shooting direction of the camera; if the camera is not directly above, but obliquely above, etc., the depth direction is the vertical line direction of the horizontal plane of the object (tray, conveyor belt, basket, etc.), or the gravity line direction, all of which are possible.

Specifically, firstly extracting a highest layer region according to a depth image, establishing a highest layer point cloud, then carrying out segmentation processing based on deep learning on the highest layer point cloud, and segmenting the highest layer point cloud and a scene point cloud of a stack type to obtain the highest layer point cloud of the stack type; and then, carrying out edge extraction processing on the highest layer point cloud of the stack type to obtain the highest layer edge point cloud.

And step S120, projecting the highest-layer edge point cloud to a two-dimensional plane to obtain an image to be identified of the target.

In the method of this embodiment, the stack type is identified based on the 2D matching method, and then the highest layer edge point cloud of the stack type needs to be projected to a two-dimensional plane, and specifically, 2D conversion is performed on the highest layer edge point cloud according to the height information of the highest layer and the internal reference information of the camera, so as to obtain the target to-be-identified image.

And step S130, performing edge matching processing according to the standard box image and the target image to be identified, and determining each box image area in the target image to be identified according to the edge matching result.

The standard box body images are preset box body image templates, and when the standard box body images are implemented, the standard box body images with different specifications can be stored in advance so as to be suitable for identifying stack types formed by the box bodies with different specifications.

Specifically, the standard box image is moved to any one possible position in the target to-be-identified image, whether the edge of the standard box image is coincident with the edge of the target to-be-identified image at the moment is judged, if so, the region in the target to-be-identified image corresponding to the standard box image, which is the box image region, is indicated that the standard box is moved to the position. Wherein, each box image area in the target image to be identified is basically that each box group of the highest layer of the stack corresponds to the image area in the target image to be identified.

In an alternative way, the edges of the color image are synthesized with the edges extracted based on the depth image, so as to improve the accuracy of identifying the stacking boxes. Specifically, a color image shot from the upper part of a stack type is obtained, and edge extraction processing is carried out on the color image to obtain a color edge image; synthesizing the color edge image and the target image to be identified to obtain a highest-layer edge synthesized image; and performing edge matching processing according to the standard box image and the highest layer edge composite image.

And step S140, determining the position information of each stacking box in the highest layer of the stacking type according to each box image area.

After the fact that each stacking box body of the highest stacking layer corresponds to the image area in the target image to be identified is identified, the spatial position information of each stacking box body of the highest stacking layer is determined, for example, the spatial position information of each stacking box body is determined through means such as coordinate conversion, so that a robot can perform unstacking operation according to the spatial position information in a subsequent process.

According to the recognition method for the box body of the stack in the stack type, the highest layer point cloud of the stack type is extracted according to the depth image shot from the upper side of the stack type, and the highest layer edge point cloud is extracted according to the highest layer point cloud of the stack type; projecting the highest layer edge point cloud to a two-dimensional plane to obtain an image to be identified of the target; performing edge matching processing according to the standard box image and the target image to be identified, and determining each box image area in the target image to be identified according to an edge matching result; and determining the position information of each group of stacked boxes in the highest layer of the stacked type according to the image areas of each box. According to the method, the highest layer point cloud is extracted through the depth image, the highest layer edge point cloud is extracted, the highest layer edge point cloud is projected to the two-dimensional plane to obtain the target to-be-identified image, the standard box body image and the target to-be-identified image are subjected to edge matching, so that the image areas of the box bodies of each stack of the highest layer are identified, and finally the physical space position information of the corresponding box bodies is positioned according to the image areas of the box bodies of each stack of the highest layer.

Fig. 2 is a flowchart illustrating a method for identifying stacking boxes in a stack according to another embodiment of the present invention, where the method of the present embodiment is applicable to a stack with the same specification for each stacking box, as shown in fig. 2, and includes the following steps:

step S210, extracting the highest layer point cloud of the stack type according to the depth image shot from the upper part of the stack type, and extracting the highest layer edge point cloud according to the highest layer point cloud of the stack type.

Firstly, extracting a highest layer region according to a depth image, and establishing a highest layer point cloud, optionally, generating the point cloud according to elements such as a laser detector, a visible light detector such as an LED, an infrared detector, a radar detector and the like. Then, carrying out segmentation processing based on deep learning on the highest-layer point cloud, and segmenting the stacked highest-layer point cloud from other scene point clouds to obtain the stacked highest-layer point cloud; and then, carrying out edge extraction processing on the highest layer point cloud of the stack type to obtain the highest layer edge point cloud.

And step S220, projecting the edge point cloud of the highest layer to a two-dimensional plane according to the height information of the highest layer and the internal reference information of the camera to obtain an image to be identified of the target.

In the method of the embodiment, recognition is performed based on a 2D matching mode, the highest-layer edge point cloud of the stack type is projected to a two-dimensional plane, and specifically, 2D conversion is performed on the highest-layer edge point cloud according to the height information of the highest layer and the internal reference information of the camera, so that a target image to be recognized is obtained.

In the implementation, due to the influence of factors such as a camera shooting angle, the shape of the target image to be identified is not necessarily a standard rectangle, and in order to improve the efficiency and accuracy of stack type identification, the shape of the target image to be identified is corrected in the mode of the embodiment. Specifically: judging whether the shape of the target image to be identified is rectangular or not; if not, correcting the target to-be-identified image to correct the shape of the target to-be-identified image into a rectangle, and obtaining a corrected target to-be-identified image. That is, if the shape of the target to-be-recognized image is not rectangular, for example, may be trapezoidal, irregular quadrangular, or the like, the shape of the target to-be-recognized image is corrected to a regular rectangle, and then edge matching processing is performed for the corrected target to-be-recognized image.

Step S230, let i=1.

And step S240, performing edge matching processing according to the standard box image and the ith wheel of to-be-identified image, and determining the image area of each ith wheel of the edge box image according to the edge matching result.

When the value of i is 1, the ith round of image to be identified is the target image to be identified or the corrected target image to be identified.

Step S250, according to any ith-wheel edge box image area, determining the position information of a target box corresponding to the ith-wheel edge box image area in the highest layer of the stack and the unstacking configuration information of the target box, so as to perform unstacking operation on the target box according to the unstacking configuration information and the position information.

Wherein, the unstacking configuration information includes: gripping position information, gripping order information, and movement direction information. In the method of the embodiment, after determining the image area of the edge box, destacking configuration information for the corresponding box is determined, so that the robot destacks the group of boxes according to the destacking configuration information.

Optionally, the moving direction information and the grabbing position information of the target box are determined according to at least one group of adjacent edges, which are overlapped with the positions of at least one group of adjacent edges of the image to be identified of the ith wheel, in the image area of the ith wheel edge box.

In this manner, the grabbing position of the box body is a box body angle, and a set of adjacent edges, which are coincident with the positions of a set of adjacent edges of the i-th wheel to be identified, in the image area of the edge box body are determined, so that the angle formed by the set of adjacent edges, which are coincident in position, in the image area of the edge box body is a grabbing angle, and the direction contained by the set of adjacent edges is the movable direction of the edge box body. For example, assuming that the edges a and B of the edge box image area are just edges of the image to be recognized, the angle formed by the edges a and B is a grippable angle, and the direction between the edges a and B opposite to the edge box image area is the direction in which the edge box is movable. In practice, if there is only one box at the highest level of the stack, then all four corners of the box are grippable.

In the prior art, the unstacking is usually performed by aligning the suction cup with the center of the box, but because the size of the unstacking clamp is fixed, when the size of the box is small relative to the clamp, if the suction cup is aligned with the center of the box, other boxes beside the box can be covered, and the box beside the box is sucked up. Different from the prior art, the mode of this embodiment is with the angle of case as snatching the position, then can guarantee to snatch at every turn and only snatch a case to, according to the edge that is located buttress type edge of case confirm the direction of movement after snatching, through above-mentioned two aspects, can avoid the process of unstacking to makeing mistakes, guarantee the accuracy of unstacking.

In practical application, after the edge matching processing of each round is finished, the position information and the unstacking configuration information of the corresponding box body can be determined according to the image area of the edge box body identified by the round, so that unstacking operation can be carried out on the box body of the highest layer of the stack type. Or, the destacking operation may be performed after all the edge matching processes are finished, in which the destacking sequence still follows the above-mentioned round sequence, that is, destacking is performed on the boxes corresponding to the image areas of the 1 st round of edge boxes, destacking is performed on the boxes corresponding to the image areas of the 2 nd round of edge boxes, and so on.

Step S260, deleting each ith round of edge box image area from the ith round of images to be identified, and judging whether the matching termination condition is met or not according to the deleted residual images; if yes, go to step S270; if not, go to step S280.

Step S270, confirming that the box image area identification is finished.

And if the residual images meet the matching termination condition, the box image region identification contained in the target image to be identified is ended. Each box image area in the target image to be identified comprises: and (5) each of the 1 st round to the i th round of edge box image areas.

And S280, i is assigned as i+1, the rest images are determined as i-th round images to be identified, and the step S240 is performed in a jumping manner.

If the remaining images do not meet the matching termination condition, the remaining images are used as images to be identified of the next round, the step is skipped to step S240, and edge matching processing is performed on the images to be identified of the next round.

In the method of the embodiment, the specific implementation mode of determining the image areas of each stacking box body in the target image to be identified through edge matching is a cyclic processing mode. A specific processing manner will be described with reference to fig. 3a and 3b, in which fig. 3a shows a stacking schematic diagram of each box of the highest layer of the stack type, as shown in fig. 3a, in which 9 boxes are stacked on the highest layer of the stack type, and fig. 3b shows a schematic diagram of an edge matching process of the target image to be recognized.

When the value of i is 1, the edge matching processing of the first round is performed. And specifically, carrying out edge matching processing according to the standard box image and the 1 st round of image to be identified (namely the target image to be identified), and determining a 1 st round of edge box image area according to an edge matching result, wherein the 1 st round of edge box image area corresponds to each group of box bodies positioned at the edge in the highest layer of the stack.

And then deleting the 1 st round edge box image area from the 1 st round of images to be identified to obtain a residual image, and judging whether the residual image meets a matching termination condition. If the matching termination condition is met, the box image area contained in the target image to be identified is identified, and the 1 st round of edge box image area is directly output; if the matching termination condition is not met, the fact that the residual image possibly contains a box image area is indicated, i is assigned to be 2, the residual image at the moment is determined to be the image to be identified in the 2 nd round, and the image to be identified in the 2 nd round is subjected to second round edge matching processing.

Referring to fig. 3b, fig. 1 in fig. 3b is the image to be identified in the 1 st round, after the first round of edge matching processing is finished, the image area including 4 edge boxes in the image to be identified in the 1 st round is identified, and referring to fig. 3b (2), that is, image areas 1, 3, 7, 9 correspond to boxes 1, 3, 7, 9 stacked at the highest layer respectively. Deleting the image area of the 1 st round of edge box body from the 1 st round of images to be identified to obtain the 2 nd round of images to be identified, and performing second round of edge matching processing, wherein the image (3) in fig. 3b is the 2 nd round of images to be identified.

When the value of i is 2, the second round of edge matching processing is performed. And (3) continuing to carry out edge matching processing on the standard box image and the 2 nd round of images to be identified, determining a 2 nd round of edge box image area in the 2 nd round of images to be identified according to a matching result, and removing the stack box bodies positioned at the edge of the rest stack boxes positioned after each stack box positioned at the edge in the highest layer, namely the stack box bodies corresponding to the 2 nd round of edge box image area.

Then, deleting the 2 nd round of edge box image areas from the 2 nd round of images to be identified, judging whether the residual images at the moment meet the matching termination condition, if so, indicating that the identification of the box image areas contained in the target images to be identified is finished, wherein each box image area in the target images to be identified comprises: wheel 1-wheel 2 edge box image area; if not, the residual image possibly contains a box image area, i is assigned to 3, the residual image after the 2 nd round of images to be identified delete each 2 nd round of edge box image area is determined to be the 3 rd round of images to be identified, and third round of edge matching processing is carried out on the 3 rd round of images to be identified.

Referring to fig. 3b, after the second round of edge matching processing is finished, the image area including 4 edge boxes in the image to be identified in the 2 nd round is identified, see (4) in fig. 3b, that is, the image areas 2, 4, 6, 8 correspond to boxes 2, 4, 6, 8 stacked at the highest layer respectively. Deleting the image area of the 2 nd round of edge box body from the 2 nd round of images to be identified to obtain the 3 rd round of images to be identified, and carrying out third round of edge matching processing, wherein the image (5) in fig. 3b is the 3 rd round of images to be identified.

When the value of i is 3, the third round of edge matching processing is performed. And continuing to perform edge matching processing on the standard box image and the image to be identified of the 3 rd round, and determining the image area of the box image of the 3 rd round in the image to be identified of the 3 rd round according to a matching result.

And then deleting the image area of the 3 rd round edge box body from the image to be identified of the 3 rd round to obtain a residual image, and judging whether the residual image meets the matching termination condition. If the matching termination condition is met, the box image area contained in the target image to be identified is identified, and each box image area in the target image to be identified comprises: each of the 1 st-3 rd edge box image areas; if the matching termination condition is not met, the fact that the residual image possibly contains a box image area is indicated, i is assigned to be 4, the residual image at the moment is determined to be a 4 th round of image to be identified, and 4 th round of edge matching processing is conducted on the 4 th round of image to be identified.

After the third round of edge matching process is finished, the image area including 1 edge box in the 3 rd round of images to be identified is identified, see (6) in fig. 3b, namely, the image area 5, which corresponds to the highest layer of stacked boxes 5. And deleting the 3 rd round of edge box image areas from the 3 rd round of images to be identified, wherein the residual area is zero, and directly outputting the 1 st round to the 3 rd round of edge box image areas after the identification of the box image areas in the target area to be identified is finished.

In an optional manner, determining whether the matching termination condition is satisfied according to the remaining image after deletion specifically includes: judging whether the area of the residual image is smaller than a preset value or not; if yes, meeting the matching termination condition; otherwise, the match termination condition is not satisfied. For example, the preset value may be an area of a standard box image. If the area of the residual image is smaller than that of the standard box image, the residual image is insufficient to form an image area corresponding to one box, and the cycle is terminated.

In an optional mode, edge matching processing is performed according to the standard box image and the ith round of to-be-identified image, and specific implementation modes of determining the image area of each ith round of edge box according to the edge matching result are as follows:

moving and traversing in the ith round of images to be identified by using the standard box body image; when the standard box image moves to any traversing position angle, calculating the edge coincidence degree of the standard box image and the ith wheel of image to be identified; and determining the image areas of the ith-wheel edge box body in the ith-wheel image to be identified according to the edge overlapping ratio.

In this manner, the movement of the standard box image includes the movement of the distance and the movement of the angle. Taking the first round of edge matching processing as an example, moving and traversing the standard box image in the 1 st round of images to be identified, calculating the coincidence degree of the edge of the standard box image and the edge of the 1 st round of images to be identified when the standard box image moves to each traversing position and each traversing angle, and finally determining the 1 st round of edge box image area in the 1 st round of images to be identified according to the coincidence degree of the edge of the standard box image and the edge of the 1 st round of images to be identified when the standard box body moves to any position and angle. For example, when the standard box image moves to the position angle a, and at this time, the edge of the standard box image coincides with the edge of the 1 st round of to-be-identified image, it can be determined that the area corresponding to the standard box image in the 1 st round of to-be-identified image is an edge box image area when the standard box image moves to the position and the angle. Referring to fig. 4, fig. 4 shows a schematic diagram of a positional relationship between a standard box image and a target to-be-identified image, and as shown in fig. 4, when the standard box image is positioned at an angle a, the region of the target to-be-identified image corresponding to the standard box is an edge box image region.

Further alternatively, the specific implementation mode for calculating the edge overlap ratio of the standard box image and the ith round of image to be identified is as follows: and calculating the edge contact ratio of any group of adjacent edges of the standard box image and any group of adjacent edges of the ith round of image to be identified. In this way, the efficiency and accuracy of the edge matching process can be improved.

In practical applications, because the accuracy of the point cloud is not high in some cases, the edges of the standard box image and the edges of the image to be identified do not completely coincide, but most of the areas coincide. Based on the above, in order to improve the accuracy of recognition, the coincidence reliability is further calculated on the basis of the coincidence, and the edge box image region is determined based on the coincidence reliability.

Specifically, according to the edge overlapping ratio, the specific implementation mode of determining the image area of each ith wheel edge box in the image to be identified of the ith wheel is as follows:

calculating the length proportion between each edge in the ith round of images to be identified and the corresponding edge of the standard box body image; calculating the coincidence credibility of the standard box image when the standard box image moves to the position angle according to the edge coincidence degree and the length proportion between each edge in the ith wheel of image to be identified and the corresponding edge of the standard box image; and determining the image areas of the ith-wheel edge box body in the ith-wheel images to be identified according to the coincidence reliability.

Referring to fig. 4, the information to be calculated includes: the method comprises the steps of calculating the coincidence reliability A of a standard box image when the position angle A is formed, optionally calculating the weighted sum to obtain the coincidence reliability according to the edge coincidence A, the length proportion A1 and the length proportion A2, and optionally calculating the weighted sum to obtain the coincidence reliability according to the edge coincidence and the preset weight and the length proportion and the preset weight thereof, wherein the coincidence A is the standard box image when the position angle A is formed, the length proportion A1 of the standard box image when the position angle A is formed, and the length proportion of the edge 2 of the standard box image when the position angle A is formed. Similarly, the three information including the edge overlapping ratio B, the length proportion B1 and the length proportion B2 are also calculated for the position angle B, and the overlapping credibility B of the standard box image when the position angle B is obtained is calculated in the same manner. And then, determining the region of the corresponding standard box image corresponding to the image to be identified in the 1 st round when the superposition reliability is highest as a1 st round edge box image region.

In an alternative way, the standard box image is utilized to move and traverse in each preset area of the ith round of images to be identified; and aiming at any preset area, determining an ith wheel edge box image area of the preset area according to the position angle information of the standard box image when the edge overlap ratio is highest.

In this method, a travel origin, a travel step length, and a travel angle of the standard box image are predetermined, and the standard box image is moved from the origin in accordance with the step length and the angle. In order to improve traversing efficiency, dividing an ith round of images to be identified into a plurality of preset areas according to a preset origin, a preset traversing step length and a preset traversing angle, traversing the standard box images in any preset area, taking the highest edge overlap ratio of the standard box images and the ith round of images to be identified as an effective result, and determining an ith round of edge box image area in the preset area according to position angle information of the standard box images when the edge overlap ratio is the highest.

For example, referring to fig. 4, the edge contact ratio a of the standard box image and the target image to be recognized at the position angle a is higher than the edge contact ratio B of the standard box image and the target image to be recognized at the position angle B, and of course, in practical implementation, there are many such position angles B that the edge contact ratio is smaller than the edge contact ratio a, which is not shown one by one. And when the standard box image is determined to move to the position angle A, the region corresponding to the standard box image in the target to-be-identified image is an edge box image region in the preset region.

In order to avoid the problem that the overall size of the target to be identified is larger due to errors in the segmentation processing process based on deep learning, after the 1 st-round edge box image areas are obtained, comparing the image areas in the 1 st-round edge box image areas with the highest-layer point cloud, and filtering at least one 1 st-round edge box image area if the area of the at least one 1 st-round edge box image area exceeding the highest-layer point cloud reaches a preset proportion. Because the highest layer point cloud is true and accurate, after each 1 st round of edge box image areas are obtained, each 1 st round of edge box image area is compared with the highest layer point cloud, if the 1 st round of edge box image areas exceed the highest layer point cloud, the 1 st round of edge box image areas are incorrect, and the 1 st round of edge box image areas are filtered.

According to the method for identifying the stacking boxes in the stack type, provided by the embodiment, the mode is adopted to inwards advance from the outer edge in a circulating edge matching mode, and each stacking box in the highest layer of the stack type is identified layer by layer, so that the accuracy of identification can be improved, and the accuracy of subsequent operation is ensured; meanwhile, the grabbing angles of the box body are positioned, the box body can be grabbed by utilizing the sucker alignment grabbing angles, so that the unstacking accuracy can be improved, and the problem of false grabbing caused by mismatching of the sizes of the clamp and the box body is avoided; in addition, the moving direction of the grabbing box body is determined according to the edge of the edge box body image area which coincides with the edge of the target image to be identified, so that the grabbing box body is ensured to move towards the barrier-free direction, and smooth unstacking is ensured.

Fig. 5 is a schematic structural view of an identification device for stacking boxes in a stack according to another embodiment of the present invention. As shown in fig. 5, the apparatus includes:

the edge extraction module 51 is adapted to extract a highest layer point cloud of the stack type according to a depth image shot from above the stack type, and extract a highest layer edge point cloud according to the highest layer point cloud of the stack type;

the conversion module 52 is adapted to project the highest layer edge point cloud to a two-dimensional plane to obtain an image to be identified of the target;

the edge matching module 53 is adapted to perform edge matching processing according to the standard box image and the target image to be identified, and determine each box image area in the target image to be identified according to the edge matching result;

and the positioning module 54 is suitable for determining the position information of each stacking box in the highest layer of the stacking type according to the image area of each box.

In an alternative, the apparatus further comprises:

the edge matching module 53 is further adapted to:

In an alternative way, the edge matching module 53 is further adapted to perform the following steps:

step S0, let i=1;

In an alternative way, the edge matching module 53 is further adapted to:

In an optional manner, each preset area is obtained by dividing according to preset origin position information, preset traversal step length information and preset traversal angle information.

In an alternative way, the edge matching module 53 is further adapted to:

In an alternative, the apparatus further comprises:

In an alternative way, the edge extraction module 51 is further adapted to:

the edge matching module 53 is further adapted to: and performing edge matching processing according to the standard box body image and the highest layer edge synthetic image.

In an alternative, the apparatus further comprises:

In an alternative, the unstacking configuration module is further adapted to:

In an alternative, the conversion module 52 is further adapted to:

According to the recognition device for the box bodies of the stack in the stack type, provided by the embodiment, the recognition device can recognize each box body of the stack in the highest layer of the stack type layer by pushing inwards from the outer edge in a circular edge matching mode, so that the recognition accuracy can be improved, and the accuracy of subsequent operation can be ensured; meanwhile, the grabbing angles of the box body are positioned, the box body can be grabbed by utilizing the sucker alignment grabbing angles, so that the unstacking accuracy can be improved, and the problem of false grabbing caused by mismatching of the sizes of the clamp and the box body is avoided; in addition, the moving direction of the grabbing box body is determined according to the edge of the edge box body image area which coincides with the edge of the target image to be identified, so that the grabbing box body is ensured to move towards the barrier-free direction, and smooth unstacking is ensured.

The embodiment of the invention provides a nonvolatile computer storage medium, which stores at least one executable instruction, and the computer executable instruction can execute the identification method of the stacking boxes in the stacking type in any method embodiment.

FIG. 6 illustrates a schematic diagram of an embodiment of a computing device of the present invention, and the embodiments of the present invention are not limited to a particular implementation of the computing device.

As shown in fig. 6, the computing device may include: a processor 602, a communication interface (Communications Interface), a memory 606, and a communication bus 608.

Wherein: processor 602, communication interface 604, and memory 606 perform communication with each other via communication bus 608. Communication interface 604 is used to communicate with network elements of other devices, such as clients or other servers. The processor 602 is configured to execute the program 610, and may specifically perform the relevant steps in the above-described embodiment of the method for identifying a stacking box in a stack of a computing device.

In particular, program 610 may include program code including computer-operating instructions.

The processor 602 may be a central processing unit CPU or a specific integrated circuit ASIC (Application Specific Integrated Circuit) or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included by the computing device may be the same type of processor, such as one or more CPUs; but may also be different types of processors such as one or more CPUs and one or more ASICs.

A memory 606 for storing a program 610. The memory 606 may comprise high-speed RAM memory or may further comprise non-volatile memory (non-volatile memory), such as at least one disk memory.

The algorithms or displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general-purpose systems may also be used with the teachings herein. The required structure for a construction of such a system is apparent from the description above. In addition, embodiments of the present invention are not directed to any particular programming language. It will be appreciated that the teachings of the present invention described herein may be implemented in a variety of programming languages, and the above description of specific languages is provided for disclosure of enablement and best mode of the present invention.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the above description of exemplary embodiments of the invention, various features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be construed as reflecting the intention that: i.e., the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the apparatus of the embodiments may be adaptively changed and disposed in one or more apparatuses different from the embodiments. The modules or units or components of the embodiments may be combined into one module or unit or component and, furthermore, they may be divided into a plurality of sub-modules or sub-units or sub-components. Any combination of all features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or units of any method or apparatus so disclosed, may be used in combination, except insofar as at least some of such features and/or processes or units are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments herein include some features but not others included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

Various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that some or all of the functionality of some or all of the components according to embodiments of the present invention may be implemented in practice using a microprocessor or Digital Signal Processor (DSP). The present invention can also be implemented as an apparatus or device program (e.g., a computer program and a computer program product) for performing a portion or all of the methods described herein. Such a program embodying the present invention may be stored on a computer readable medium, or may have the form of one or more signals. Such signals may be downloaded from an internet website, provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third, etc. do not denote any order. These words may be interpreted as names. The steps in the above embodiments should not be construed as limiting the order of execution unless specifically stated.

Claims

1. A method for identifying a stacking box in a stack type comprises the following steps:

extracting the highest layer point cloud of the stack type according to a depth image shot from above the stack type, and extracting the highest layer edge point cloud according to the highest layer point cloud of the stack type;

projecting the highest-layer edge point cloud to a two-dimensional plane to obtain a target image to be identified;

the edge matching processing is performed on the standard box image and the target image to be identified, each box image area in the target image to be identified is determined according to an edge matching result, and the method further comprises the following steps:

step S0, let i=1;

step S1, carrying out edge matching processing according to a standard box image and an ith wheel of to-be-identified image, and determining the image area of each ith wheel of the edge box according to an edge matching result; wherein, the step S1 further includes: moving and traversing in the ith round of images to be identified by utilizing the standard box body image; when the standard box body image moves to any traversing position angle, calculating the edge coincidence degree of the standard box body image and the ith wheel of image to be identified; according to the edge overlapping ratio, determining the image area of each ith wheel of the box body at the edge of the ith wheel of the image to be identified;

When the value of i is 1, the ith round of image to be identified is the target image to be identified;

step S2, deleting the image areas of the ith round of edge box body from the ith round of images to be identified, and judging whether the matching termination condition is met or not according to the deleted residual images; if yes, confirming that the identification of the box body image area is finished; if not, assigning i as i+1, determining the residual image as an i-th round of image to be identified, and jumping to execute the step S1; if the area of the residual image is smaller than a preset value, a matching termination condition is met; if the area of the residual image is not smaller than the preset value, the matching termination condition is not satisfied;

determining position information of each group of stacked boxes in the highest layer of the stacked type according to each box image area, wherein each box image area comprises: and (3) identifying the 1 st round to the i th round of edge box image areas at the end of the box image area.

2. The method of claim 1, wherein the method further comprises:

judging whether the shape of the target image to be identified is rectangular or not;

if not, correcting the target to-be-identified image to correct the shape of the target to-be-identified image into a rectangle, so as to obtain a corrected target to-be-identified image;

The performing edge matching processing according to the standard box image and the target to-be-identified image further comprises:

3. The method of claim 1, wherein the calculating the edge overlap of the standard bin image and the i-th round of images to be identified further comprises:

and calculating the edge coincidence degree of any group of adjacent edges of the standard box body image and any group of adjacent edges of the ith round of image to be identified.

4. The method of claim 1, wherein the moving through the ith round of images to be identified with the standard bin image further comprises:

moving and traversing in each preset area of the ith round of images to be identified by utilizing the standard box images;

and determining the image area of each ith wheel edge box in the image to be identified of the ith wheel according to the edge overlapping ratio further comprises:

5. The method of claim 4, wherein the respective preset areas are divided according to preset origin position information, preset traverse step information, and preset traverse angle information.

6. The method of claim 1, wherein the determining each ith wheel edge bin image area in the ith wheel image to be identified according to edge overlap further comprises:

calculating the coincidence reliability of the standard box image when the standard box image moves to the position angle according to the edge coincidence degree and the length proportion between each edge in the ith wheel of image to be identified and the corresponding edge of the standard box image;

and determining the image areas of the box body at the edge of each ith wheel in the images to be identified of the ith wheel according to the coincidence reliability.

7. The method of claim 1, wherein the method further comprises:

and comparing each 1 st-round edge box body image area with the highest-layer point cloud, and filtering at least one 1 st-round edge box body image area if the area of the at least one 1 st-round edge box body image area exceeding the area of the highest-layer point cloud reaches a preset proportion.

8. The method of claim 1, wherein the method further comprises:

and performing edge matching processing according to the standard box image and the highest layer edge composite image.

9. The method of claim 1, wherein after determining each ith round of edge bin image areas from the edge matching results, the method further comprises:

wherein the unstacking configuration information includes: and capturing position information and moving direction information.

10. The method of claim 9, wherein determining unstacking configuration information for a target bin corresponding to any i-th wheel-edge bin image region according to the i-th wheel-edge bin image region further comprises:

And determining the moving direction information and the grabbing position information of the target box according to at least one group of adjacent edges, which are overlapped with the positions of at least one group of adjacent edges of the image to be identified of the ith wheel, in the image area of the box at the ith wheel.

11. The method of claim 1, wherein the projecting the highest layer edge point cloud to a two-dimensional plane further comprises:

12. An identification device for a stack box in a stack type, comprising:

the conversion module is suitable for projecting the highest-layer edge point cloud to a two-dimensional plane to obtain an image to be identified of the target;

the edge matching module is suitable for executing the following steps:

step S0, let i=1;

the positioning module is suitable for determining the position information of each group of stacked boxes in the highest layer of the stacked type according to each box image area, wherein each box image area comprises: and (3) identifying the 1 st round to the i th round of edge box image areas at the end of the box image area.

13. The apparatus of claim 12, wherein the apparatus further comprises:

the correction module is suitable for judging whether the shape of the target image to be identified is rectangular; if not, correcting the target to-be-identified image to correct the shape of the target to-be-identified image into a rectangle, so as to obtain a corrected target to-be-identified image;

The edge matching module is further adapted to:

14. The apparatus of claim 12, wherein the edge matching module is further adapted to:

15. The apparatus of claim 12, wherein the edge matching module is further adapted to:

16. The apparatus of claim 15, wherein the respective preset regions are partitioned according to preset origin position information, preset traverse step information, and preset traverse angle information.

17. The apparatus of claim 12, wherein the edge matching module is further adapted to:

18. The apparatus of claim 12, wherein the apparatus further comprises:

and the filtering module is suitable for comparing each 1 st round of edge box body image area with the highest layer point cloud, and filtering at least one 1 st round of edge box body image area if the area of the at least one 1 st round of edge box body image area exceeding the area of the highest layer point cloud reaches a preset proportion.

19. The apparatus of claim 12, wherein the edge extraction module is further adapted to:

the edge matching module is further adapted to: and performing edge matching processing according to the standard box image and the highest layer edge composite image.

20. The apparatus of claim 12, wherein the apparatus further comprises:

the unstacking configuration module is suitable for determining unstacking configuration information of a target box corresponding to any ith wheel edge box image area according to any ith wheel edge box image area so as to perform unstacking operation on the target box according to the unstacking configuration information; wherein the unstacking configuration information includes: and capturing position information and moving direction information.

21. The apparatus of claim 20, wherein the unstacking configuration module is further adapted to:

22. The apparatus of claim 12, wherein the conversion module is further adapted to:

23. A robotic system comprising the identification device of a palletized bin in a stack of any one of claims 12-22.

24. A computing device, comprising: the device comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete communication with each other through the communication bus;

the memory is configured to store at least one executable instruction, where the executable instruction causes the processor to perform operations corresponding to the method for identifying a stacking box in a stack according to any one of claims 1 to 11.

25. A computer storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the method of identifying palletized boxes in a palletized form as claimed in any one of claims 1 to 11.