CN113253737A

CN113253737A - Shelf detection method and device, electronic equipment and storage medium

Info

Publication number: CN113253737A
Application number: CN202110682472.XA
Authority: CN
Inventors: 胡立志; 卢维; 王政; 李铭
Original assignee: Zhejiang Huaray Technology Co Ltd
Current assignee: Zhejiang Huaray Technology Co Ltd
Priority date: 2021-06-21
Filing date: 2021-06-21
Publication date: 2021-08-13
Anticipated expiration: 2041-06-21
Also published as: CN113253737B

Abstract

The invention provides a goods shelf detection method and device, electronic equipment and a storage medium, wherein the method comprises the following steps: the mobile robot acquires point cloud data of the reflected laser; determining the orientation of the goods shelf based on a point cloud cluster formed by clustering point cloud data; respectively determining shelf leg groups meeting the size of the shelf as candidate shelf leg groups; selecting a shelf leg group close to the preset shelf coordinate from the candidate shelf leg groups, and utilizing the central coordinate and the orientation of the shelf leg in the selected shelf leg group; converting the central coordinate of the shelf into a world coordinate system to obtain the position of a detected target point; and the mobile robot drives into the lower part of the goods shelf according to the position of the target point, lifts the goods shelf and transports the goods shelf to the destination. The invention can accurately identify the center position of the goods shelf and can accurately operate.

Description

Shelf detection method and device, electronic equipment and storage medium

Technical Field

The present invention relates to shelf detection technologies for mobile robots, and in particular, to a shelf detection method and apparatus, an electronic device, and a storage medium.

Background

In recent years, the rapid development of mobile robots in the industrial field has strongly pushed the progress of industry 4.0. Compared with an industrial robot, the mobile robot has more flexibility, can independently complete operation in a certain scene environment, greatly liberates productivity, and has extremely high application value in the fields of manufacturing industry, logistics industry and the like.

When a mobile robot performs a task operation, such as object handling in a warehouse, the mobile robot needs to accurately know the position of itself in the current environment in the changed environment, so that the task can be accurately performed. In addition, the mobile robot needs to accurately determine information such as the position and the pose of the goods shelf so as to accurately perform related operation on the goods shelf.

In the technical scheme of determining the position and the pose of the shelf at present, whether the shelf is positioned on the shelf leg is determined by using characteristic points of laser reflected on the shelf leg, and a plurality of interference discrete points are often arranged on the metal shelf leg, so that the shelf leg is mistakenly considered not to be a shelf characteristic and is mistakenly considered not to be detected. Due to the existence of the interference points, the estimated center of the goods shelf deviates from the actual center, so that the mobile robot cannot be guaranteed to accurately lift the actual center position of the goods shelf, and cannot accurately enter one side of the goods shelf to perform related operations.

Disclosure of Invention

The invention provides a shelf detection method and device, electronic equipment and a storage medium, which are used for at least solving the technical problems in the prior art.

The invention provides a shelf detection method on one hand, which comprises the following steps:

after the mobile robot determines that the mobile robot reaches the shelf identification area, laser scanning is carried out to obtain point cloud data of reflected laser;

filtering the point cloud data based on the size of the goods shelf, and clustering the filtered point cloud data;

filtering the clustered point cloud data based on the size of the goods shelf and preset pose information thereof;

determining a first orientation of the shelf based on a point cloud cluster formed by clustering the filtered point cloud data; determining a first center coordinate of a shelf leg based on a first orientation of the shelf using point cloud data on the shelf leg as an observation;

sequentially traversing all the goods shelf legs, taking at least two goods shelf legs as a group, and respectively determining the goods shelf leg group meeting the size of the goods shelf as a candidate goods shelf leg group;

selecting a shelf leg group with a difference value smaller than a first set threshold value from the candidate shelf leg groups, using a second central coordinate of a shelf leg in the selected shelf leg group as an observation value, and estimating a third central coordinate and a second orientation of the shelf based on the first central coordinate;

converting the third center coordinate of the shelf into a world coordinate system according to the relevant position parameters of the scanning laser and the current pose of the mobile robot to obtain the position of the detected target point;

and the mobile robot is switched to a milemeter navigation mode, drives into the lower part of the goods shelf according to the position of the target point, lifts the goods shelf and transports the goods shelf to a destination.

Optionally, before lifting the rack and transporting the rack to the destination, the method further comprises:

starting an upward-looking camera of the mobile robot, scanning an identification code on a goods shelf, adjusting the position of the mobile robot by detecting the relative pose of the upward-looking camera relative to the identification code until the relative pose is smaller than a second set threshold value, stopping the position adjustment of the mobile robot, recording the relative pose, and calling a workbench to lift the goods shelf.

Optionally, the filtering the point cloud data based on the size of the shelf includes:

and deleting the point cloud data when the distance between the laser point cloud and the preset shelf center is determined to be larger than a first set distance.

Optionally, the filtering the clustered point cloud data includes:

deleting the point cloud cluster when the distance between two points farthest from each other in the point cloud clusters gathered as one type is larger than a second set distance; or

Deleting the point cloud clusters when the number of the point clouds in the point cloud clusters gathered as a class is smaller than a first preset number or larger than a second preset number; or

And acquiring a fourth center coordinate of each point cloud cluster from the point cloud clusters gathered into one type, and deleting the point cloud cluster when judging that the distance between the fourth center coordinate and the preset center position of the shelf is a third set distance.

Optionally, the determining, with at least two shelf legs as a group, a shelf leg group in which a shelf size is satisfied is determined, respectively, as a candidate shelf leg group, including:

optionally selecting three of the four different shelf legs as a group, judging whether the difference between the distance between the two shelf legs and the actual length of the shelf in the group is smaller than a third set threshold, if so, judging whether the difference between the distance between the two shelf legs and the actual width of the shelf in the group is smaller than a fourth set threshold, if so, judging whether the two shelf legs with the length are vertical to the two shelf legs with the width, if so, judging whether the difference between the hypotenuse of a triangle formed by the length and the width and the actual length of the hypotenuse of the shelf is smaller than a fifth set threshold, and if so, verifying the central coordinates of the shelf corresponding to the three shelf legs, and determining that the three shelf legs meet the position relation of the actual shelf legs through verification;

if any three shelf legs of the four different shelf legs meet the position relation of the actual shelf legs, the four shelf legs are four legs of the shelf;

continuously traversing the shelf legs except the determined four shelf legs, taking the three shelf legs as a group, judging whether the difference between the distance between the two shelf legs and the actual length of the shelf in the group is less than a third set threshold value or not, if so, judging whether the difference between the distance between the two shelf legs and the actual width of the shelf in the group is smaller than a fourth set threshold value or not, if so, judging whether the two shelf legs with the length are vertical to the two shelf legs with the width, if so, then, it is determined whether the difference between the hypotenuse of the triangle formed by the length and the width and the actual length of the hypotenuse of the shelf is smaller than a fifth set threshold, and if so, checking the center coordinates of the goods shelf corresponding to the three goods shelf legs, wherein the three goods shelf legs meet the position relation of the actual goods shelf legs through the checking, and the three goods shelf legs in the group are three legs of the goods shelf;

and continuously traversing the shelf legs except the determined four or three shelf legs, taking the two shelf legs as a group, estimating and checking the orientation of the shelf and the center coordinate of the shelf when the difference between the distance between the two shelf legs and the actual length of the shelf is determined to be less than a third set threshold value, and determining that the two shelf legs in the group meet the position relation of the actual shelf legs by checking, wherein the two shelf legs in the group are the two legs of the shelf.

Optionally, the determining a first orientation of the shelf based on the point cloud clusters formed by the filtered point cloud data clustering comprises:

when the point cloud number in each point cloud cluster is judged to exceed a third preset number, carrying out Principal Component Analysis (PCA) decomposition on the point cloud cluster, and solving an initial value of the orientation of the goods shelf under a laser coordinate system to serve as a first orientation of the goods shelf;

and when the point cloud number in the point cloud cluster is detected to be smaller than the third preset number, determining the first orientation of the shelf by utilizing the closeness degree of the length between two shelf legs in the shelf legs and the actual length or the actual width of the shelf.

In another aspect, the present invention provides a shelf detecting apparatus, including:

the first acquisition unit is used for carrying out laser scanning after determining that the goods shelf identification area is reached, and acquiring point cloud data of reflected laser;

the clustering unit is used for filtering the point cloud data based on the size of the goods shelf and clustering the filtered point cloud data;

the filtering unit is used for filtering the clustered point cloud data based on the size of the goods shelf and preset pose information thereof;

the first determining unit is used for determining a first orientation of the goods shelf based on the point cloud clusters formed by clustering the filtered point cloud data; determining a first center coordinate of a shelf leg based on a first orientation of the shelf using point cloud data on the shelf leg as an observation;

the second determining unit is used for sequentially traversing all the goods shelf legs, taking at least two goods shelf legs as a group, and respectively determining the goods shelf leg group meeting the size of the goods shelf as a candidate goods shelf leg group;

an estimation unit configured to select a shelf leg group having a difference value smaller than a first set threshold value from among the candidate shelf leg groups and a preset shelf coordinate, and estimate a third center coordinate and a second orientation of the shelf based on the first center coordinate using a second center coordinate of a shelf leg in the selected shelf leg group as an observation value;

the second acquisition unit is used for converting the third center coordinate of the shelf into a world coordinate system according to the relevant position parameter of the scanning laser and the current pose of the mobile robot to obtain the position of the detected target point;

and the control unit is used for switching the mobile robot into a milemeter navigation mode, driving the mobile robot into the lower part of the goods shelf according to the position of the target point, lifting the goods shelf and transporting the goods shelf to a destination.

Optionally, the control unit is further configured to:

before lifting the goods shelf and transporting the goods shelf to the destination, starting an upward-looking camera of the mobile robot, scanning an identification code on the goods shelf, adjusting the position of the mobile robot by detecting the relative pose of the upward-looking camera relative to the identification code until the relative pose is smaller than a second set threshold, stopping the position adjustment of the mobile robot, recording the relative pose, and calling a workbench to lift the goods shelf.

Optionally, the clustering unit is further configured to:

Optionally, the filtering unit is further configured to:

Optionally, the second determining unit is further configured to:

Optionally, the first determining unit is further configured to:

The invention provides an electronic device, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus; a memory for storing a computer program; and the processor is used for realizing the steps of the shelf detection method when executing the program stored in the memory.

Yet another aspect of the present invention provides a computer readable storage medium having a computer program stored therein, which when executed by a processor implements the steps of the shelf detection method.

The invention does not need to limit the laser scanning range in advance, and can correctly estimate the center of the goods shelf even if the laser is biased; according to the method, the characteristics of the shelf legs are not required to be identified, an error equation is constructed by using the point clouds on the shelf legs as observed values to solve the center of the shelf legs, different shelf center methods are designed by combining the number of candidate shelf legs, and the shelf legs of a target shelf can be accurately screened when false detection occurs, the shelf legs are shielded by 1 to 2, and a plurality of shelves are put together, so that the correct shelf is identified, the center of the shelf is calculated, and the operation is accurately performed. The invention can detect the orientation of the goods shelf, ensure that the mobile robot enters the lower part of the goods shelf from one side of the goods shelf according to the required direction and has an attempt mechanism for the failure of goods shelf identification. According to the invention, after the central position of the goods shelf is identified, the navigation mode is switched to the odometer navigation mode, so that the positioning error caused by the interference of laser during butt joint is avoided. The mobile robot is combined with the upward-looking camera, so that the mobile robot can be close to the center of the goods shelf as much as possible when the goods shelf is lifted. The upward-looking camera not only plays a role in recording the position of the mobile robot relative to the goods shelf, but also utilizes the two-dimensional code on the back surface of the goods shelf identified by the upward-looking camera to perform relative navigation and adjust the position and posture of the mobile robot, so that the mobile robot is close to the center of the goods shelf as much as possible, the influence of interference points on the position and posture estimation of the center of the goods shelf caused by the detection of legs of the goods shelf is eliminated, the follow-up goods shelf is guaranteed to be moved to another target point, and the risk that the goods shelf is in obstacle is reduced. Meanwhile, the upward-looking camera ensures that the mobile robot can be adjusted to the orientation relative to the goods shelf after entering the lower part of the goods shelf.

Drawings

FIG. 1 shows a flow diagram of a shelf detection method of an embodiment of the invention;

fig. 2 is a schematic view showing an overall structure of a mobile robot according to an embodiment of the present invention;

FIG. 3 shows a schematic view of a shelf model of an embodiment of the invention;

FIG. 4 is a schematic diagram of a specific implementation of the shelf detection method according to the embodiment of the invention;

FIG. 5 is a schematic diagram illustrating the structure of a shelf detection device according to an embodiment of the present invention;

fig. 6 shows a block diagram of an electronic device of an embodiment of the invention.

Detailed Description

In order to make the objects, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Fig. 1 shows a flowchart of a shelf detection method according to an embodiment of the present invention, and as shown in fig. 1, the shelf detection method according to the embodiment of the present invention includes the following processing steps:

step 101, after the mobile robot determines that the mobile robot reaches a shelf identification area, laser scanning is carried out, and point cloud data of reflected laser are obtained.

In the embodiment of the invention, in the navigation of the mobile robot, a topological map is stored in the mobile robot and is used for marking a moving route of the mobile robot and target points which the mobile robot needs to pass through and reach, and meanwhile, a goods shelf area is marked to show that goods shelves need to be transported to the ideal position; when the goods shelf is detected, the mobile robot firstly navigates through the odometer and the laser radar, and then the mobile robot reaches a goods shelf identification preparation point. The odometer can preliminarily estimate the angle and distance variation of the mobile robot, and a common wheeled encoder can estimate the pose of the robot at the current moment according to the pose variation at the previous moment and the pose at the previous moment. The two-dimensional laser radar can acquire a sensor of two-dimensional plane information and is used for detecting two-dimensional plane profile information of the surrounding environment. After the goods shelf legs are accurately detected through the navigation laser, the real position of the center of the goods shelf is estimated, and the goods shelf is accurately found nearby a goods shelf target point to be lifted by using a milemeter navigation mode.

And 102, filtering the point cloud data based on the size of the shelf, and clustering the filtered point cloud data.

In the embodiment of the invention, when the distance between the laser point cloud and the preset shelf center is determined to be greater than the first set distance, the point cloud data is deleted. That is, when it is determined that the distance between the point cloud in the laser point cloud and the other point clouds is too large, it is obvious that the laser point cloud is not reflected on the shelf leg but may be an interference point cloud, which may be deleted.

And clustering the filtered point cloud data, namely aggregating the point clouds possibly forming the goods shelf as one class so as to conveniently determine the goods shelf.

And 103, filtering the clustered point cloud data based on the size of the goods shelf and preset pose information thereof.

In the embodiment of the invention, in the point cloud clusters gathered into one type, when the distance between two points farthest away is greater than a second set distance, the point cloud cluster is deleted; or

104, determining a first orientation of a goods shelf based on a point cloud cluster formed by clustering filtered point cloud data; a first center coordinate of a shelf leg is determined based on a first orientation of the shelf using the point cloud data on the shelf leg as an observation.

Wherein, the first orientation of goods shelves is confirmed to the point cloud cluster that forms based on the point cloud data cluster that filters includes: when the point cloud number in each point cloud cluster is judged to exceed a third preset number, carrying out Principal Component Analysis (PCA) decomposition on the point cloud cluster, and solving an initial value of the orientation of the goods shelf under a laser coordinate system to serve as a first orientation of the goods shelf; and when the point cloud number in the point cloud cluster is detected to be smaller than the third preset number, determining the first orientation of the shelf by utilizing the closeness degree of the length between two shelf legs in the shelf legs and the actual length or the actual width of the shelf.

And 105, traversing all the goods shelf legs in sequence, taking at least two goods shelf legs as a group, and respectively determining a goods shelf leg group meeting the size of a goods shelf as a candidate goods shelf leg group.

Determine candidate shelf leg group, specifically include: optionally selecting three of the four different shelf legs as a group, judging whether the difference between the distance between the two shelf legs and the actual length of the shelf in the group is smaller than a third set threshold, if so, judging whether the difference between the distance between the two shelf legs and the actual width of the shelf in the group is smaller than a fourth set threshold, if so, judging whether the two shelf legs with the length are vertical to the two shelf legs with the width, if so, judging whether the difference between the hypotenuse of a triangle formed by the length and the width and the actual length of the hypotenuse of the shelf is smaller than a fifth set threshold, and if so, verifying the central coordinates of the shelf corresponding to the three shelf legs, and determining that the three shelf legs meet the position relation of the actual shelf legs through verification;

And 106, selecting a shelf leg group with a difference value smaller than a first set threshold value from the candidate shelf leg groups, using a second center coordinate of a shelf leg in the selected shelf leg group as an observation value, and estimating a third center coordinate and a second orientation of the shelf based on the first center coordinate.

In the embodiment of the invention, after the candidate shelf leg group is determined, the center coordinates of the shelf are verified based on the candidate shelf leg to determine whether the candidate shelf leg group is close to the preset shelf center of the shelf, and further determine whether the shelf leg group is an actual shelf.

And step 107, converting the third center coordinate of the shelf into a world coordinate system according to the relevant position parameters of the scanning laser and the current pose of the mobile robot, so as to obtain the position of the detected target point.

In the embodiment of the invention, after the actual shelf leg group is determined, the center coordinate of the shelf is converted into the world coordinate system, so that the mobile robot can move to the corresponding position according to the coordinate of the shelf, and the operation such as moving the shelf is performed.

And step 108, switching the mobile robot into a milemeter navigation mode, driving the mobile robot into the lower part of the goods shelf according to the position of the target point, lifting the goods shelf and transporting the goods shelf to a destination.

In the embodiment of the invention, in the process of carrying the goods shelf by the mobile robot and walking, the pose of the mobile robot is estimated through laser navigation, and the relative variation of the mobile robot relative to the identification code is superposed, so that the goods shelf can follow an ideal route in the moving process of the mobile robot, and the goods shelf can be placed at an ideal target point when the goods shelf is put down. In the embodiment of the invention, the identification code can be a two-dimensional code.

The essence of the technical solution of the embodiment of the present invention is further clarified by specific examples below.

The shelf detection method provided by the embodiment of the invention can estimate the pose of the mobile robot based on the odometer, the laser sensor and the prior map, when the shelf is butted, all shelf legs meeting the conditions are detected, then the central position of the shelf is estimated according to the shelf legs and the position of the shelf is converted into a world coordinate system, then the mobile robot is switched into an odometer navigation mode to enter the lower part of the shelf, when the shelf is nearly reached, the mobile robot opens the upper view camera, scans and detects the identification codes such as two-dimensional codes attached to the lower part of the shelf, and when the two-dimensional codes are detected, the mobile robot is switched into a two-dimensional code navigation mode, so that the central position of the shelf which can reach by the mobile robot is ensured, and the shelf is successfully lifted.

Fig. 2 is a schematic diagram illustrating an overall structure of a mobile robot according to an embodiment of the present invention, and as shown in fig. 2, the mobile robot according to the embodiment of the present invention is a robot based on a odometer and a 2D lidar sensor, and mainly includes a mobile ground, a 2D lidar sensor, and an upward-looking camera. The mobile ground plate comprises a motion controller, a motor, a battery, an embedded computer, a milemeter and the like. The mobile robot can perform high-precision positioning in a two-dimensional grid map generated based on a warehousing environment. The map comprises a goods shelf, a workbench, a house support frame, goods needing to be carried and the like. The mobile robot uses a odometer to estimate the amount of change in the robot motion while scanning the environment profile using a 2D lidar. Before the robot enters the goods shelf, the goods shelf is identified at a goods shelf identification preparation point, the central position of the goods shelf is detected, the goods shelf is switched into a mileometer mode to walk into the lower part of the goods shelf, and then the two-dimensional code below the goods shelf is scanned by using the upward-looking camera, so that the mobile robot walks to the position right below the goods shelf to successfully lift the goods shelf. And the corresponding shelf model is shown in fig. 3, wherein the zero degree direction of the shelf is determined by the direction of the two-dimensional code pasted on the back of the shelf.

Fig. 4 shows a schematic diagram of a specific implementation of the shelf detection method according to the embodiment of the present invention, and details of steps of the shelf detection according to the embodiment of the present invention are described below with reference to the processing flow shown in fig. 4:

the detection of the shelf legs is performed, and the center of the shelf legs and the orientation of the shelf are estimated.

In the robot navigation, a topological map is used for marking a moving route of the mobile robot and target points which the robot needs to pass through and reach, and meanwhile, a goods shelf area is marked to indicate that goods shelves need to be transported to the ideal position; when the goods shelf is detected, the robot firstly navigates through the odometer and the laser radar to ensure thatThe mobile robot reaches a shelf identification preparation point; length l of the legs of the upper and lower delivery racks_wAnd width l_hOr shelf leg radius, shelf length L and width w and the desired location P of the shelf and the desired shelf angle in the map. In the embodiment of the invention, the detection method is detailed for common rectangular shelf legs as cases, and the detection of the circular shelf legs is similar to the method, and only corresponding error variance needs to be changed.

And reading the 2D laser point cloud, and preprocessing the laser point cloud. Firstly, filtering all point clouds under a laser radar coordinate system, and rejecting the point clouds when detecting that the distance between the laser point clouds and a preset shelf center P is larger than a certain distance d1 (for example, d1 is set as the longest edge of a double shelf), wherein the point clouds are not points on shelf legs. And then clustering the point clouds, traversing from the point cloud with the index of 0, and when the distance between two adjacent laser point clouds is smaller than a certain threshold (such as the longest edge of two times of a shelf leg), regarding the two point clouds to be hit on the same object and be clustered into one class until the adjacent point clouds which do not meet the conditions are hit, and regarding the two point clouds not to be the same class point cloud, and performing next clustering.

Because the goods shelf legs are generally made of metal materials, a plurality of interference points are formed on the goods shelf legs through laser, the influence of the interference points is not eliminated, the estimated position deviates from the center of the actual goods shelf too far, the mobile robot cannot be lifted at the center of the goods shelf, and even the mobile robot cannot scan the two-dimensional code on the back of the goods shelf after entering the lower part of the goods shelf. At this time, it is determined whether the distance between two points farthest from each other among the points grouped into one category is greater than a certain threshold (for example, 5 times the longest side of the shelf leg), and if the distance is greater than the threshold, it is determined that the point cloud cluster is not a point cloud cluster hit on the shelf leg or a point cloud cluster interfering with a too large shelf leg, and the point cloud cluster is directly deleted. Meanwhile, judging whether the number of the point clouds in each point cloud cluster meets a certain number requirement, limiting the thicknesses of the shelf legs, and deleting the clustered point clouds which do not meet the number requirement, wherein the number of the point clouds hit on the shelf legs cannot be too large or too small; and solving a central coordinate value ci of each point cloud cluster, judging whether the distance between the central coordinate value and the ideal shelf central position P issued or configured by the network is less than d1 or not, and deleting the point cloud clusters with the distance greater than d 1. The cluster of point clouds that remains is likely to be a detected shelf leg.

Judging whether the number of the point clouds in each point cloud cluster is enough, such as more than 6 point clouds, and if so, performing Principal Component Analysis (PCA) decomposition on the point cloud cluster to obtain an initial value of the orientation of the shelf under a laser coordinate system. The specific mode is as follows:

to pair

After Singular Value Decomposition (SVD), the eigenvector corresponding to the maximum eigenvalue is the principal direction of the point cloud cluster

. Wherein the content of the first and second substances,

clustering a certain point cloud in the point cloud cluster; the | Z | is the number of a cluster of clustering point cloud points; the initial value of the shelf orientation angle is:

=

–arctan(l_h/l_w) （3）

will be provided with

Limited to plus or minus 180 degrees. When the number of point clouds in the point cloud cluster is detected to be insufficient, such as less than 6 or less than 3 point clouds, c in two shelf legs is utilized_iAnd c_jTo estimate the shelf orientation, if c_iAnd c_jA distance d between_ijAnd the length L of the goods shelf issued or configured by the network is close to the length L of the goods shelf, the initial value of the orientation of the goods shelf is as follows:

= arctan((c_jy-c_iy)/(c_jx-c_ix)) （4）

if d is_ijClose to the delivered shelf width w, the initial value of the shelf orientation is:

= arctan(cjy-ciy)/(cjx-cix))+pi/2 (5)

limiting

Is between plus and minus 180 degrees.

Suppose the shelf leg center coordinate to be estimated is c_lWith the shelf oriented as

Then, an error equation ei is constructed using the point cloud hit on the shelf leg as follows:

ei = si((p_jx-c_lx)*cos(

)+(p_jy-c_ly)*sin(

)-l_h/2) (6)

and si is the weight of each viewpoint point cloud in the residual error item, and the error variance of observation construction of all point clouds on the shelf legs is subjected to nonlinear optimization solution, so that the central coordinate and the shelf orientation of each candidate shelf leg can be solved.

And screening the candidate shelf legs, and solving the center of the shelf by using the correct shelf legs.

In goods shelves butt joint environment, more than one goods shelves that often deposit, mobile robot can detect a plurality of goods shelves legs, and goods shelves are placed the slope of different degrees occasionally or skew storehouse position point by the manual work, simultaneously, because sheltering from of object in the environment, often can not detect four goods shelves legs of a goods shelves, detect the goods shelves leg this moment, select correct goods shelves leg to ask goods shelves center. The method comprises the following specific steps:

traversing all the rack legs in sequence, with four different rack legs in a group, means that four rack legs (l) can be detected at the same time this time_i1,l_i2,l_i3,l_i4) Then, in the four rack legs, every three are in a group, and whether two rack legs (such as l) exist in the group is judged_i1、l_i2) The distance between the shelf and the shelf is close to the length L of the shelf; judging whether the distance between the two shelf legs (li1, li3) is close to the width of the shelf or not; if the line segment exists, whether the edge formed by the li1 and the li2 is perpendicular to the edge formed by the li1 and the li3 is continuously judged, if the line segment is perpendicular to the edge formed by the li2 and the li3, whether the edge formed by the li2 and the li3 is close to the length of the inclined edge of the shelf is continuously judged, if the conditions are met, whether the li3 is above or below the line segment formed by the li1 and the li2 is judged again, and if the li3 is above the line segment, the center coordinate of the shelf is updated as follows:

C_i= (p_li1+p_li2) + R*w/2 (7)

meanwhile, the three shelf legs corresponding to the marks are divided into a lower left side, a lower right side and an upper left side of the shelf. If li3 is below the line segment, then the center coordinates of the shelf are updated as:

C_i = (p_li1+p_li2) - R*w/2 (8)

meanwhile, the three shelf legs corresponding to the marks are divided into an upper left side, an upper right side and a lower left side of the shelf.

Wherein Ci is the estimated shelf leg center of the group of shelf legs; p_lixThe coordinates corresponding to the shelf legs lix and R are the rotation matrix corresponding to the shelf orientation, as shown in equation (11).

The orientation estimate of the pallet is estimated using the lower left, lower right or upper left and upper right pallet leg coordinates, and the orientation estimate in this set of cases is:

= arctan((p_li1y-p_li2y)/( p_li1x-p_li2x)) (10)

R=

（11）

similarly, the three groups are verified (li1, li2, li4), (li2, li3, li4) and (li1, li3, li4), if the three groups are verified, the four shelf legs are considered to be four legs of the shelf, the four legs are recorded, and the estimated value of the center of the shelf in the group is updated, as shown in formula 12. And traversing the shelf legs of the candidate shelf leg set next time, and directly skipping if the four shelf legs are recorded in a group, so that the searching speed is increased.

C_i=(p_li1+p_li2+p_li3+p_li4)/4 (12)

And continuously traversing the shelf legs in the candidate shelf legs, taking three shelf legs as a group, and directly skipping if the id of the three shelf legs is a subset of the four shelf legs in the group. Otherwise, the three shelf legs are taken as a group to be stored and recorded. Two of the shelf legs (t) are judged_j1、t_j2) Is close to the length of the goods shelf, and the two goods shelf legs are (t)_j1、t_j3) Whether or not it is close to the width of the goods shelf, and t_j1、t_j2Edge of composition and t_j1、t_j3Perpendicular to the constituent sides, t_j2、t_j3Whether the distance of (a) is close to the length of the bevel edge of the goods shelf. If both are satisfied, the method of estimating the center position of the shelf and the orientation of the shelf is similar to the aforementioned equations (7) (8), and three shelves are respectively corresponding to the lower left, lower right, upper right and upper left positions of the shelf.

The traversal continues through the shelf legs of the candidate shelf legs, with two of the shelf legs in a group, skipping directly if the id of the two shelf legs (rk1, rk2) is above a subset of a group of four shelf legs or a group of three shelf legs. Otherwise, judging whether the lengths of the two goods shelf legs are similar to the length of the goods shelf, if so, the estimated value of the orientation of the goods shelf is as follows:

= arctan((p_rk1y-p_rk2y)/(p_rk1x-p_rk2x)) （13）

if the preset shelf center is above the line segment formed by rk1 and rk2, the estimated value of the shelf center is:

C_k=(p_rk1+p_rk2)+R*w/2 （14）

and marking the two shelf legs as lower left and lower right, otherwise the shelf center estimate is:

C_k= (p_rk1+p_rk2)-R*w/2 （15）

the two shelf legs are marked upper left and upper right. If the distance between the two shelf legs is close to the width of the shelf, the shelf orientation estimate is:

= arctan((p_rk1y-p_rk2y)/(p_rk1x-p_rk2x))+pi/2 （16）

if the preset shelf center is to the right of rk1 and rk2 segments, the shelf center estimate is: c_k=(p_rk1+p_rk2)+R*l/2 （17）

Marking two shelf legs as upper left and lower left, otherwise the shelf center estimate is:

Ck = (p_rk1+p_rk2)-R*l/2 （18）

two or attritors are labeled upper right and lower right. If the distance between the two shelf legs is close to the slant length of the shelf, the estimated value of the shelf orientation is as follows:

= arctan((p_rk1y-p_rk2y)/(p_rk1x-p_rk2x))-arctan(l_h/l_w) （19）

the estimated value of the center of the shelf at this time is:

C_k = (p_rk1y + p_rk1y)/2 （20）

limiting the angle of the goods shelf between plus and minus 180 degrees, judging the goods shelf center estimated by the plurality of groups of goods shelf legs, storing the nearest goods shelf leg id from the preset goods shelf center, and recording the number of the goods shelf legs. For the case that the number of the shelf legs is 2, due to the large degree of shelf offset, only depending on the relative position relationship of the preset shelf position relative to the shelf leg line, an incorrect shelf position may be calculated, and the actual position may be at the symmetrical position of the line segment. Therefore, the mobile robot needs to continuously identify at intervals of time or distance in the process of butting the goods shelves, and three candidate goods shelf legs are ensured to ensure the accuracy of goods shelf position estimation; or when the mobile robot runs to the target point, the two-dimensional code below the goods shelf is not scanned all the time, the mobile robot retreats to the goods shelf identification point to restart the goods shelf identification, and when the mobile robot goes to the target point, the mobile robot goes to another symmetrical position if the two-dimensional code is not scanned again.

Assuming that the shelf center needs to be solved as C, the initial value is the estimated value obtained as described above. Keeping the shelf leg ids of the next group as id0, id1, id2 and id3, assuming that four ids correspond to the lower left, lower right, upper right and upper left of the shelf respectively (there may be only three shelf legs or two shelf legs in the id, taking four shelf legs as an example), the error equation constructed by id0 is:

u_id0x = c_x –cos(

)*l/2+sin(

)*w/2 (21)

u_id0y = c_y– sin(

)*l/2-cos(

)*w/2 (22)

the corresponding error terms are:

e_id0x= |c_lid0x – u_id1x| (23)

e_id0y = |c_lid0y-u_id1y| (24)

wherein, cx, cy and

x, y direction coordinates and orientation of the center point of the shelf to be found, and clid0 represents the previously found center coordinates of the shelf leg. The error equations of other vertexes are constructed similarly without id0, and the error equations of the shelf legs are combined and solved by nonlinear optimization to obtain the coordinates and the orientation of the center point of the shelf. And comparing the orientation of the goods shelf with the orientation of the goods shelf issued on the upper layer, adding pi/2, pi and 3pi/2 to the current orientation respectively, comparing which direction is the closest to the orientation of the issued goods shelf in the four directions, and returning the direction as the butt joint direction when the mobile robot enters the goods shelf.

And (4) butting shelves of the mobile robot.

After the center of the goods shelf is solved, the center of the goods shelf is converted into a world coordinate system through external parameters related to laser and the current pose of the robot to obtain a detected target point Z, then the mobile robot is switched into a mileometer navigation mode and enters the position below the goods shelf, and when the target point Z is reached, an upward-looking camera is opened to search a two-dimensional code below the goods shelf. The goods shelf legs are generally made of metal materials, interference points exist when laser is generally applied to the metal goods shelf legs, the estimated goods shelf center has deviation, at the moment, when an upward-looking camera scans the two-dimensional code on the back face of the goods shelf, the position of the mobile robot is adjusted by detecting the relative pose of the camera relative to the two-dimensional code, the relative pose is guaranteed to be smaller than a certain threshold value, the mobile robot is stopped to adjust and record the relative pose, the mobile robot is considered to reach the position under the goods shelf center, and the goods shelf is lifted. In the process that the mobile robot backs up the goods shelf to walk, the relative variation of the mobile robot relative to the two-dimensional code is superposed through the robot pose estimated through laser navigation, so that the goods shelf can follow an ideal route in the moving process of the mobile robot, and the goods shelf can be placed at an ideal target point when the goods shelf is put down.

The invention does not need to limit the laser scanning range in advance, and can correctly estimate the center of the goods shelf even if the laser is biased; according to the method, the characteristics of the shelf legs are not required to be identified, an error equation is constructed by using the point clouds on the shelf legs as observed values to solve the center of the shelf legs, different shelf center methods are designed by combining the number of candidate shelf legs, and the shelf legs of a target shelf can be accurately screened when false detection occurs, the shelf legs are shielded by 1 to 2, and a plurality of shelves are put together, so that the correct shelf is identified, the center of the shelf is calculated, and the operation is accurately performed. The invention can detect the orientation of the goods shelf, ensure that the mobile robot enters the lower part of the goods shelf from one side of the goods shelf according to the required direction and has an attempt mechanism for the failure of goods shelf identification. According to the invention, after the central position of the goods shelf is identified, the navigation mode is switched to the odometer navigation mode, so that the positioning error caused by the interference of laser during butt joint is avoided. The mobile robot is combined with the upward-looking camera, so that the mobile robot can be close to the center of the goods shelf as much as possible when the goods shelf is lifted. The upward-looking camera not only plays a role in recording the position of the mobile robot relative to the goods shelf, but also utilizes the two-dimensional code on the back surface of the goods shelf identified by the upward-looking camera to perform relative navigation and adjust the position and posture of the mobile robot, so that the mobile robot is close to the center of the goods shelf as much as possible, the influence of interference points on the position and posture estimation of the center of the goods shelf caused by the detection of legs of the goods shelf is eliminated, the follow-up goods shelf is guaranteed to be moved to another target point, and the risk that the goods. Meanwhile, the upward-looking camera ensures that the mobile robot can be adjusted to the orientation relative to the goods shelf after entering the lower part of the goods shelf.

Fig. 5 is a schematic diagram illustrating a composition structure of a shelf detection apparatus according to an embodiment of the present invention, and as shown in fig. 5, the shelf detection apparatus according to the embodiment of the present invention includes:

the first acquisition unit 50 is used for performing laser scanning after determining that the goods shelf identification area is reached, and acquiring point cloud data of reflected laser;

a clustering unit 51, configured to filter the point cloud data based on the size of the shelf, and cluster the filtered point cloud data;

the filtering unit 52 is used for filtering the clustered point cloud data based on the size of the shelf and preset pose information thereof;

a first determining unit 53 for determining a first orientation of the shelf based on the point cloud cluster formed by clustering the filtered point cloud data; determining a first center coordinate of a shelf leg based on a first orientation of the shelf using point cloud data on the shelf leg as an observation;

a second determining unit 54, configured to sequentially traverse all shelf legs, and determine, as candidate shelf leg groups, shelf leg groups satisfying the shelf size, with at least two shelf legs as a group;

an estimating unit 55 configured to select, among the candidate shelf leg groups, a shelf leg group having a difference value from a preset shelf coordinate smaller than a first set threshold value, and estimate a third center coordinate and a second orientation of the shelf based on the first center coordinate using a second center coordinate of a shelf leg in the selected shelf leg group as an observed value;

a second obtaining unit 56, configured to convert the third center coordinate of the shelf into a world coordinate system according to a relevant position parameter of the scanning laser and a current pose of the mobile robot, so as to obtain a detected target point position;

and the control unit 57 is used for switching the mobile robot into an odometer navigation mode, driving the mobile robot into the lower part of the goods shelf according to the position of the target point, lifting the goods shelf and transporting the goods shelf to a destination.

As an implementation manner, the control unit 57 is further configured to:

As an implementation manner, the clustering unit 51 is further configured to:

As an implementation manner, the filtering unit 52 is further configured to:

As an implementation manner, the second determining unit 54 is further configured to:

As an implementation manner, the first determining unit 53 is further configured to:

In an exemplary embodiment, the first obtaining Unit 50, the clustering Unit 51, the filtering Unit 52, the first determining Unit 53, the second determining Unit 54, the estimating Unit 55, the second obtaining Unit 56, the control Unit 57, and the like may be implemented by one or more Central Processing Units (CPUs), Graphics Processing Units (GPUs), Baseband Processors (BPs), Application Specific Integrated Circuits (ASICs), a Digital Signal Processor (DSP), a Programmable Logic Device (PLD), a Complex Programmable Logic Device (CPLD), a Field Programmable Gate Array (FPGA), a general purpose Processor, a Controller, a Microcontroller (MCU), a Microprocessor (Microprocessor), or other electronic components, for performing the steps of the shelf detection method of the foregoing embodiments.

In the embodiment of the present disclosure, the specific manner in which each unit in the shelf inspecting apparatus shown in fig. 5 performs operations has been described in detail in the embodiment related to the method, and will not be described in detail here.

Next, an electronic apparatus 11 according to an embodiment of the present application is described with reference to fig. 6.

As shown in fig. 6, the electronic device 11 includes one or more processors 111 and memory 112.

The processor 111 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 11 to perform desired functions.

Memory 112 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, Random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, Read Only Memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium and executed by processor 111 to implement the authentication methods of the various embodiments of the present application described above and/or other desired functions. Various contents such as an input signal, a signal component, a noise component, etc. may also be stored in the computer-readable storage medium.

In one example, the electronic device 11 may further include: an input device 113 and an output device 114, which are interconnected by a bus system and/or other form of connection mechanism (not shown).

The input device 113 may include, for example, a keyboard, a mouse, and the like.

The output device 114 may output various information including the determined distance information, direction information, and the like to the outside. The output devices 114 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, among others.

Of course, for the sake of simplicity, only some of the components of the electronic device 11 relevant to the present application are shown in fig. 6, and components such as buses, input/output interfaces, and the like are omitted. In addition, the electronic device 11 may include any other suitable components, depending on the particular application.

In addition to the above-described methods and apparatus, embodiments of the present application may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to the various embodiments of the present application described in the "exemplary methods" section of this specification, above.

The computer program product may be written with program code for performing the operations of embodiments of the present application in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.

Furthermore, embodiments of the present application may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform steps in a method according to various embodiments of the present application described in the "exemplary methods" section above of this specification.

The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The foregoing describes the general principles of the present application in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present application are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present application. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the foregoing disclosure is not intended to be exhaustive or to limit the disclosure to the precise details disclosed.

The block diagrams of devices, apparatuses, systems referred to in this application are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".

It should also be noted that in the devices, apparatuses, and methods of the present application, the components or steps may be decomposed and/or recombined. These decompositions and/or recombinations are to be considered as equivalents of the present application.

The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the application to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.

Claims

1. A shelf detection method, the method comprising:

2. The method of claim 1, wherein prior to lifting the rack and transporting the rack to the destination, the method further comprises:

3. The method of claim 1, wherein the filtering the point cloud data based on a size of a shelf comprises:

4. The method of claim 1, wherein the filtering the clustered point cloud data comprises:

5. The method of claim 1, wherein the determining, as candidate shelf leg groups, shelf leg groups in which shelf sizes are satisfied, respectively, with at least two shelf legs as a group, comprises:

6. The method of claim 1, wherein determining a first orientation of a shelf based on point cloud clusters formed by the filtered point cloud data clustering comprises:

7. A shelf detection apparatus, the apparatus comprising:

8. The apparatus of claim 7, wherein the control unit is further configured to:

9. The apparatus of claim 7, wherein the clustering unit is further configured to:

10. The apparatus of claim 7, wherein the filter unit is further configured to:

11. The apparatus of claim 7, wherein the second determining unit is further configured to:

12. The apparatus of claim 7, wherein the first determining unit is further configured to:

13. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus; a memory for storing a computer program; a processor for implementing the steps of the shelf detection method according to any one of claims 1-6 when executing a program stored in the memory.

14. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the shelf detection method according to any one of claims 1-6.