WO2023226676A2

WO2023226676A2 - Unmanned forklift truck high shelf deliver method, apparatus, and device, and storage medium

Info

Publication number: WO2023226676A2
Application number: PCT/CN2023/091273
Authority: WO
Inventors: 陈文成; 吕朝顺; 黄金勇
Original assignee: 劢微机器人科技(深圳)有限公司
Priority date: 2022-05-25
Filing date: 2023-04-27
Publication date: 2023-11-30
Also published as: CN115018895A

Description

Unmanned forklift high-level shelf loading methods, devices, equipment and storage media

This application claims priority to the Chinese patent application with application number 202210577015.9 filed on May 25, 2022, the entire content of which is incorporated into this application by reference.

Technical field

The present application relates to the field of automatic control technology, and in particular to an unmanned forklift high-level shelf loading method, device, equipment and storage medium.

Background technique

With the introduction of Industry 4.0 and intelligent manufacturing, the industrial field continues to develop from traditional manufacturing to digital, intelligent, and unmanned. In an era when everything can be intelligent, unmanned technology continues to impact our attention. In smart warehousing The demand for unmanned forklift applications in the industry is increasing. Because the forklift mast is extended too high, it causes the mast to shake, and the deviation in the cargo delivery truck is too large, resulting in inaccurate cargo delivery and prone to safety accidents.

The above content is only used to assist in understanding the technical solutions of the present application, and does not represent an admission that the above content is prior art.

technical problem

The main purpose of this application is to provide an unmanned forklift high-level shelf loading method, device, equipment and storage medium, aiming to solve the problem that the existing forklift mast is too high, which can easily lead to the mast shaking and deviation in the loading truck. If it is too large, it will lead to inaccurate delivery and technical problems that may easily lead to safety accidents.

Technical solutions

In order to achieve the above purpose, this application provides a method for unmanned forklift high-level shelf loading, which method includes the following steps:

Obtain image information of forklift pallets and shelves through the depth camera on the fork arm of the unmanned forklift;

Determine the point cloud data of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf;

Determine the spatial position information of the forklift pallet and the shelf according to the point cloud data;

Determine the distance difference between the forklift pallet and the front surface of the shelf based on the spatial position information;

The fork arm is controlled to adjust its posture according to the distance difference to complete the delivery of goods.

In one embodiment, determining the point cloud data of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf includes:

Determine the point cloud information of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf;

Match the point cloud information of the forklift pallets and shelves with the preset checkerboard grid to obtain a matching three-dimensional point cloud set;

Determine the rotation matrix and translation matrix from the depth camera to the unmanned forklift body according to the matching three-dimensional point set;

The point cloud data of the forklift pallet and the shelf are determined according to the rotation matrix, the translation matrix and the point cloud information of the forklift pallet and the shelf.

In one embodiment, determining the point cloud data of the forklift pallet and the shelf based on the rotation matrix, the translation matrix and the point cloud information of the forklift pallet and the shelf includes:

Determine the camera coordinate system point cloud data of the forklift pallet and the shelf based on the point cloud information of the forklift pallet and the shelf;

Convert the camera coordinate system point cloud data into forklift coordinate system point cloud data according to the rotation matrix and the translation matrix;

The forklift coordinate system point cloud data is used as the point cloud data of the forklift pallet and shelf.

In one embodiment, the point cloud information of the forklift pallets and shelves is matched with a preset checkerboard pattern to obtain a matching three-dimensional point cloud set, including:

Determine the grayscale images of the forklift pallet and the shelf based on the point cloud information of the forklift pallet and the shelf;

Determine the checkerboard corner points based on the grayscale images of the forklift pallets and shelves;

The checkerboard plane formula is fitted based on the grayscale images of the forklift pallets and shelves and the checkerboard corner points;

Search and match points on the grayscale images of the forklift pallets and shelves according to the checkerboard plane formula to obtain a two-dimensional point cloud set with the same abscissa and ordinate;

Determine a homography matrix based on the two-dimensional point cloud collection;

A matching three-dimensional point cloud set is determined according to the homography matrix and the checkerboard plane formula.

In one embodiment, determining the point cloud data of forklift pallets and shelves based on the forklift coordinate system point cloud data includes:

Determine the point cloud image corresponding to the grayscale image of the forklift pallet and the shelf in the forklift coordinate system based on the forklift coordinate system point cloud data;

Determine the region of interest on the point cloud image corresponding to the grayscale image of the forklift pallet and the shelf in the forklift coordinate system based on the point cloud image corresponding to the grayscale image of the forklift pallet and the shelf in the forklift coordinate system;

Traverse the area of interest to obtain point cloud data of forklift pallets and shelves.

In one embodiment, determining the spatial position information of the forklift pallet and the shelf based on the point cloud data includes:

Perform discrete point filtering, normal vector filtering, point cloud smoothing, and point cloud clustering on the point cloud data to segment the target point cloud data of the pallet legs and the shelf;

Use the RANSAC algorithm on the target point cloud data to obtain point cloud plane information;

Determine the surface point cloud mean data of the forklift pallet and the surface point cloud data of the shelf based on the point cloud plane information;

The spatial position information of the forklift pallet and the shelf is determined based on the surface point cloud mean data and the surface point cloud data.

In one embodiment, the control of the fork arm to perform posture adjustment based on the distance difference to complete the delivery of goods includes:

Compare the distance difference with the pose adjustment threshold to obtain a comparison result;

When the comparison result is that the distance difference is greater than the posture adjustment threshold, control the fork arm to perform posture adjustment according to the distance difference;

When the comparison result is that the distance difference is less than or equal to the posture adjustment threshold, the posture of the fork arm is not adjusted.

In addition, in order to achieve the above purpose, this application also proposes an unmanned forklift high-level shelf loading device. The unmanned forklift high-level shelf loading device includes:

The image acquisition module is used to obtain image information of forklift pallets and shelves through the depth camera on the fork arm of the unmanned forklift;

A point cloud extraction module is used to determine the point cloud data of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf;

A position determination module, configured to determine the spatial position information of the forklift pallet and the shelf based on the point cloud data;

A difference calculation module, configured to determine the distance difference between the forklift pallet and the front surface of the shelf based on the spatial position information;

The adjustment module for placing goods is used to control the fork arm to adjust the position and posture according to the distance difference to complete the placement of goods.

In addition, in order to achieve the above purpose, this application also proposes an unmanned forklift high-level shelf loading equipment. The unmanned forklift high-level shelf loading equipment includes: a memory, a processor, and a storage device that is stored in the memory and can be used in the An unmanned forklift high shelf stocking program running on the processor, the unmanned forklift high shelf stocking program is configured to implement the steps of the unmanned forklift high shelf stocking method as described above.

In addition, in order to achieve the above purpose, this application also proposes a storage medium on which an unmanned forklift high-level shelf loading program is stored. When the unmanned forklift high-level shelf loading program is executed by the processor, the above implementation is implemented. The steps of the described unmanned forklift high shelf loading method.

beneficial effects

This application obtains the image information of the forklift pallet and the shelf through the depth camera on the fork arm of the unmanned forklift; determines the point cloud data of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf; determines the point cloud data based on the point cloud data. The spatial position information of the forklift pallet and the shelf; determining the distance difference between the forklift pallet and the front surface of the shelf according to the spatial position information; controlling the fork arm to adjust the posture according to the distance difference , to complete the delivery. In this way, the spatial position information of the forklift pallet and the shelf is determined based on the image information of the forklift pallet and the shelf collected by the depth camera on the fork arm, thereby determining the distance difference between the forklift pallet and the front surface of the shelf, thereby Adjust the position and posture of the fork arm according to the distance difference to achieve accurate delivery. This can reduce the occurrence of the mast shaking due to lifting too high when the forklift is placing goods on the high shelf, making the delivery of goods by unmanned forklifts more accurate. , reduce the possibility of safety accidents and improve user experience.

Description of the drawings

Figure 1 is a schematic structural diagram of the unmanned forklift high-level shelf placement equipment of the hardware operating environment involved in the embodiment of the present application;

Figure 2 is a schematic flow chart of the first embodiment of the unmanned forklift high shelf loading method of the present application;

Figure 3 is a schematic diagram of the installation position of the depth camera in one embodiment of the unmanned forklift high shelf loading method of the present application;

Figure 4 is a schematic flow chart of the second embodiment of the unmanned forklift high-level shelf loading method of the present application;

Figure 5 is a structural block diagram of the first embodiment of the unmanned forklift high shelf loading device of the present application.

The realization of the purpose, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings.

Embodiments of the invention

It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

Referring to Figure 1, Figure 1 is a schematic structural diagram of the unmanned forklift high-level shelf placement equipment of the hardware operating environment involved in the embodiment of the present application.

As shown in Figure 1 , the unmanned forklift high-shelf loading equipment may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize connection communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). The user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

Those skilled in the art can understand that the structure shown in Figure 1 does not constitute a limitation on the unmanned forklift high-level shelf loading equipment, and may include more or less components than shown in the figure, or some components may be combined or different. component layout.

As shown in Figure 1, memory 1005, which is a storage medium, may include an operating system, a network communication module, a user interface module, and an unmanned forklift high shelf placement program.

In the unmanned forklift high-level shelf placement equipment shown in Figure 1, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; in this application, the unmanned forklift high-level shelf placement The processor 1001 and the memory 1005 in the device can be set in the unmanned forklift high shelf stocking equipment. The unmanned forklift high shelf stocking equipment calls the unmanned forklift high shelf stocking program stored in the memory 1005 through the processor 1001. , and execute the unmanned forklift high-level shelf loading method provided by the embodiment of this application.

An embodiment of the present application provides a method for placing goods on a high-level shelf with an unmanned forklift. Refer to Figure 2 , which is a schematic flow chart of a method for placing goods on a high-level shelf with an unmanned forklift according to the first embodiment of the present application.

In this embodiment, the unmanned forklift high-level shelf loading method includes the following steps:

Step S10: Obtain the image information of the forklift pallet and shelf through the depth camera on the fork arm of the unmanned forklift.

It should be noted that the execution subject of this embodiment is a controller, which may be a processor of an unmanned forklift, a control unit, or other equipment that can realize this function, which is not limited in this embodiment.

It should be understood that unmanned forklifts are currently widely used in industry for intelligent cargo management and placement in warehouses or other scenarios. However, when unmanned forklifts are often placing goods, the forklift mast may be stretched too high. The mast will shake, which will lead to excessive deviation when placing goods and inaccurate placement, which will easily lead to safety issues such as overturning of goods and falling of goods from high altitudes. The solution in this embodiment installs depth cameras on both sides of the fork arm of the unmanned forklift to calculate the position and distance difference between the forklift pallet and the shelf based on the images collected by the depth camera, thereby accurately adjusting the posture and achieving a more precise position. For safe and accurate unmanned forklift high-level shelf placement.

In a specific implementation, two depth cameras are provided on the fork arm of the unmanned forklift. As shown in Figure 3, the installation locations of the depth cameras are specifically the left side of the left fork arm and the right side of the right fork arm of the unmanned forklift. Among them, the depth camera refers to the TOF camera, which continuously sends light pulses to the target, and then uses the sensor to receive the light returned from the object, and obtains the target distance by detecting the round-trip time of the light pulse.

It should be noted that the image information of forklift pallets and shelves refers to the specific information of images collected by two depth cameras respectively.

Step S20: Determine the point cloud data of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf.

It should be understood that the forklift pallet is the storage pallet above the fork arm of the unmanned forklift, and the shelf is the shelf where the goods need to be placed.

In a specific implementation, determining the point cloud data of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf means: obtaining the grayscale image of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf, and then based on the forklift pallet Checkerboard annotation is performed with the grayscale image of the shelf to obtain the three-dimensional point cloud data of the forklift pallet and shelf.

Step S30: Determine the spatial position information of the forklift pallet and the shelf according to the point cloud data.

It should be noted that the spatial location information refers to the specific coordinates and occupied positions of the forklift pallets and shelves in the spatial location and other related information.

It should be understood that determining the spatial position information of the forklift pallet and the shelf based on the point cloud data means: filtering, smoothing and other operations on the point cloud data, and then using the RANSAC algorithm, that is, the point cloud data can be An algorithm is used to filter out effective data, and finally the surface point cloud mean data of the forklift pallet and the surface point cloud data of the shelf are obtained, thereby determining the spatial position information of the forklift pallet and the shelf.

Further, in order to obtain accurate spatial position information, step S30 includes: performing discrete point filtering, normal vector filtering, point cloud smoothing, and point cloud clustering on the point cloud data to segment the pallet legs and the shelf. Target point cloud data; use the RANSAC algorithm on the target point cloud data to obtain point cloud plane information; determine the surface point cloud mean data of the forklift pallet and the surface point cloud data of the shelf based on the point cloud plane information ; Determine the spatial position information of the forklift pallet and the shelf according to the surface point cloud mean data and the surface point cloud data.

In the specific implementation, discrete point filtering and normal vector filtering are both filtering steps, that is, denoising the point cloud data, and then performing point cloud smoothing and clustering to obtain the point cloud data of the area between the pallet legs and the shelf. , which is the target point cloud data.

It should be noted that using the RANSAC algorithm on the target point cloud data to obtain point cloud plane information means: using the RASNSAC algorithm on the target point cloud data, that is, effectively filtering the target point cloud data to remove abnormal data and noise. , the point cloud plane information is obtained, that is, the point cloud data of the forklift's pallet legs and the plane of the shelf.

It should be understood that determining the surface point cloud mean data of the forklift pallet based on the point cloud plane information and the surface point cloud data of the shelf means: re-extracting the point cloud plane information to obtain the forklift pallet The average z coordinate of the point cloud data on the pallet surface is the average height position of the plane where the forklift pallet is located. The surface point cloud data of the shelf is also the point cloud data of the plane where the shelf is located.

In this way, filtering and denoising based on point cloud data are implemented, and then the point cloud data of the plane corresponding to the forklift pallet and the shelf is obtained, thereby accurately obtaining the spatial position information of the forklift pallet and the shelf, and then accurately determining the forklift The adjustment position of the fork arm.

Step S40: Determine the distance difference between the forklift pallet and the front surface of the shelf according to the spatial position information.

In a specific implementation, determining the distance difference between the forklift pallet and the front surface of the shelf based on the spatial location information refers to: based on the z coordinate of the surface point cloud mean data of the forklift pallet in the spatial location information. Data, and the z-coordinate data of the surface point cloud data of the shelf, find the difference d, which is the value to be adjusted for the fork arm, that is, the distance difference.

Step S50: Control the fork arm to adjust its posture according to the distance difference to complete the delivery of goods.

It should be noted that controlling the position and posture adjustment of the fork arm according to the distance difference to complete the delivery of goods means determining the posture range of the fork arm that needs to be adjusted and the position of the target based on the distance difference, thereby completing the delivery of goods.

Further, in order to reduce the number of adjustments of the fork arm as much as possible, step S50 includes: comparing the distance difference with the pose adjustment threshold to obtain a comparison result; when the comparison result is that the distance difference is greater than the position adjustment threshold, When the posture adjustment threshold is reached, the fork arm is controlled to perform posture adjustment according to the distance difference; when the comparison result is that the distance difference is less than or equal to the posture adjustment threshold, the posture of the fork arm is not adjusted. .

It should be understood that the comparison result refers to the comparison result of the relationship between the distance difference and the posture adjustment threshold, where the posture adjustment threshold is an arbitrary numerical threshold preset by the user, and this embodiment does not be restricted.

In a specific implementation, when the comparison result is that the distance difference is greater than the posture adjustment threshold, controlling the fork arm to perform posture adjustment according to the distance difference means that when the distance difference is greater than the posture adjustment threshold, At the threshold, it is determined that the fork arm needs to be adjusted, so the fork arm of the unmanned forklift is adjusted according to the distance difference so that the position of the fork arm can safely place the goods on the shelf.

It should be noted that when the comparison result is that the distance difference is less than or equal to the posture adjustment threshold, not adjusting the posture of the fork arm means: when the distance difference is less than or equal to the posture adjustment threshold When the threshold is reached, the data used to determine the height difference between the fork arm and the shelf does not affect the safety and accuracy of placing goods, so the goods can be placed directly without adjusting the position of the fork arm.

In this way, no adjustment is made when the position difference between the fork arm and the shelf is within the allowable error range, thereby reducing unnecessary posture adjustments and making the process of placing goods on the unmanned forklift simpler and faster. .

In this embodiment, the image information of the forklift pallet and the shelf is obtained through the depth camera on the fork arm of the unmanned forklift; the point cloud data of the forklift pallet and the shelf is determined based on the image information of the forklift pallet and the shelf; the point cloud data is determined based on the point cloud data. The spatial position information of the forklift pallet and the shelf; determining the distance difference between the forklift pallet and the front surface of the shelf according to the spatial position information; controlling the position and posture of the fork arm according to the distance difference Adjust to complete the delivery. In this way, the spatial position information of the forklift pallet and the shelf is determined based on the image information of the forklift pallet and the shelf collected by the depth camera on the fork arm, thereby determining the distance difference between the forklift pallet and the front surface of the shelf, thereby Adjust the position and posture of the fork arm according to the distance difference to achieve accurate delivery. This can reduce the occurrence of the mast shaking due to lifting too high when the forklift is placing goods on the high shelf, making the delivery of goods by unmanned forklifts more accurate. , reduce the possibility of safety accidents and improve user experience.

Referring to Figure 4, Figure 4 is a schematic flowchart of a second embodiment of an unmanned forklift high-level shelf loading method of the present application.

Based on the above-mentioned first embodiment, the unmanned forklift high-level shelf loading method of this embodiment includes in step S20:

Step S201: Determine the point cloud information of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf.

It should be understood that determining the point cloud information of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf means: performing grayscale processing based on the characteristics of the depth camera based on the image information of the forklift pallet and the shelf, and then The relevant information of the point cloud data of the forklift pallets and shelves in the coordinate system of the depth camera is obtained.

Step S202: Match the point cloud information of the forklift pallets and shelves with the preset checkerboard grid to obtain a matching three-dimensional point cloud set.

It should be noted that the preset checkerboard refers to the initial checkerboard set by the user, which is used to match the grayscale images of forklift pallets and shelves to obtain a matching three-dimensional point cloud set. Among them, the matching three-dimensional point cloud set refers to a set of points with the same x and y coordinates in the grayscale images of forklift pallets and shelves.

Further, in order to accurately obtain the matching three-dimensional point cloud set, step S202 includes: determining the grayscale image of the forklift pallet and the shelf based on the point cloud information of the forklift pallet and the shelf; determining based on the grayscale image of the forklift pallet and the shelf. Checkerboard corner points; fit the checkerboard plane formula according to the grayscale image of the forklift pallet and the shelf and the checkerboard corner points; fit the grayscale image of the forklift pallet and the shelf according to the checkerboard plane formula Search and match the points to obtain a two-dimensional point cloud set with the same abscissa and ordinate; determine the homography matrix according to the two-dimensional point cloud set; determine the matching according to the homography matrix and the checkerboard plane formula A collection of 3D point clouds.

It should be noted that determining the grayscale image of the forklift pallet and the shelf based on the point cloud information of the forklift pallet and the shelf means: first extracting the grayscale image of the corresponding forklift pallet and shelf position based on the point cloud information of the forklift pallet and the shelf. degree image.

It should be understood that determining the checkerboard corner points based on the grayscale image of the forklift pallet and the shelf means: determining the checkerboard corner point of the preset checkerboard grid on the grayscale image based on the grayscale image of the forklift pallet and the shelf, That is to say, the corner points of the checkerboard corresponding to the four preset checkerboards are determined on the grayscale image of the forklift pallet and shelf.

It should be noted that fitting the checkerboard plane formula based on the grayscale images of forklift pallets and shelves means: fitting the checkerboard plane formulas in two depth camera coordinate systems, and then combining these two planes. Formula as a checkerboard plane formula.

It should be understood that searching and matching points on the grayscale images of the forklift pallets and shelves according to the checkerboard plane formula to obtain a two-dimensional point cloud set with the same abscissa and ordinate means: according to the checkerboard The plane formula searches and matches each point on the grayscale image of the forklift pallet and shelf. Points with the same x and y coordinates corresponding to the checkerboard are used as a set to obtain several two-dimensional point cloud sets.

In specific implementation, the homography matrix refers to the projection matrix between real physical coordinates and ideal pixel points. Determining a matching three-dimensional point cloud set according to the homography matrix and the checkerboard plane formula means: obtaining the corresponding three-dimensional coordinates through the homography matrix H and the two-dimensional coordinates of the checkerboard corner points on the image. xy value, that is, a matching three-dimensional point cloud set is obtained.

In this way, the checkerboard annotation method is specifically used to obtain a two-dimensional point cloud set from the points on the grayscale images of the forklift pallets and shelves, and then the two-dimensional point cloud sets are matched to obtain a matching three-dimensional point cloud set.

Step S203: Determine the rotation matrix and translation matrix from the depth camera to the unmanned forklift body according to the matching three-dimensional point set.

It should be understood that the rotation matrix is a matrix that has the effect of changing the direction of the vector but not changing the size when multiplying a vector and maintaining the properties, while the translation matrix refers to the matrix used for calculation when implementing matrix translation. And the rotation matrix and translation matrix will not change the original matrix.

Step S204: Determine the point cloud data of the forklift pallet and the shelf according to the rotation matrix, the translation matrix and the point cloud information of the forklift pallet and the shelf.

In a specific implementation, determining the point cloud data of the forklift pallet and the shelf according to the rotation matrix and the translation matrix means: completing the calibration of the point cloud data according to the rotation matrix and the translation matrix, thereby obtaining the point cloud of the forklift pallet and the shelf. data.

Further, in order to accurately obtain the point cloud data of the forklift pallet and the shelf, step S204 includes: determining the camera coordinate system point cloud data of the forklift pallet and the shelf according to the point cloud information of the forklift pallet and the shelf; The rotation matrix and the translation matrix convert the camera coordinate system point cloud data into the forklift coordinate system point cloud data; the point cloud data of the forklift pallet and the shelf are determined based on the forklift coordinate system point cloud data.

It should be noted that determining the camera coordinate system point cloud data of the forklift pallet and the shelf based on the point cloud information of the forklift pallet and the shelf refers to: determining each of the forklift pallet and the shelf based on the point cloud information of the forklift pallet and the shelf. The point cloud data of the position of the part in the coordinate system of the depth camera is used as the point cloud data of the camera coordinate system.

It should be understood that converting the camera coordinate system point cloud data into the forklift coordinate system point cloud data according to the rotation matrix and the translation matrix means: converting the point cloud on the camera coordinate system through the rotation matrix and the translation matrix. The data is converted to the forklift coordinate system, that is, each point cloud data is converted into the coordinate system, and the result is the point cloud data in the forklift coordinate system.

In a specific implementation, determining the point cloud data of the forklift pallet and the shelf based on the forklift coordinate system point cloud data means: determining the corresponding grayscale images of the forklift pallet and the shelf in the forklift coordinate system based on the forklift coordinate system point cloud data. Point cloud image, and then determine the area of interest, thereby traversing the area of interest to obtain point cloud data of forklift pallets and shelves.

In this way, the point cloud data of forklift pallets and shelves can be accurately determined through coordinate system conversion.

Further, in order to be able to determine accurate point cloud data based on the rotation matrix and the translation matrix, the step of determining the point cloud data of the forklift pallet and the shelf based on the point cloud data of the forklift coordinate system includes: The data determines the point cloud data of the forklift pallet and the shelf, including: determining the corresponding point cloud image of the grayscale image of the forklift pallet and the shelf in the forklift coordinate system according to the point cloud data of the forklift coordinate system; The point cloud image corresponding to the grayscale image of the shelf in the forklift coordinate system determines the area of interest on the point cloud image corresponding to the grayscale image of the forklift pallet and the shelf in the forklift coordinate system; traverse the area of interest to obtain the forklift pallet Point cloud data with shelves.

It should be noted that on the point cloud image corresponding to the grayscale image of the forklift pallet and shelf in the forklift coordinate system, the distribution image of the point cloud in each image area is determined based on the rotation matrix and the translation matrix, and then a ROI area is drawn. It is the area of interest. It traverses the area of interest to obtain point cloud data on the point cloud image, and finally obtains accurate point cloud data of forklift pallets and shelves.

In this way, it is possible to accurately filter out the area of interest and the point cloud data of forklift pallets and shelves from the corresponding point cloud images in the forklift coordinate system from the grayscale images of forklift pallets and shelves, making subsequent calculations more accurate. And reduce the amount of data processing.

In this embodiment, the point cloud information of the forklift pallet and the shelf is determined based on the image information of the forklift pallet and the shelf; the point cloud information of the forklift pallet and the shelf is matched with a preset checkerboard pattern to obtain a matching three-dimensional point cloud Set; determine the rotation matrix and translation matrix from the depth camera to the unmanned forklift body according to the matching three-dimensional point set; determine the point cloud information of the forklift pallet and shelf according to the rotation matrix, the translation matrix and the forklift pallet Determine point cloud data for forklift pallets and racks. In this way, the point cloud information of the forklift pallet and the shelf is obtained based on the image information of the forklift pallet and the shelf, and then matched and calibrated with the preset checkerboard to obtain a matching three-dimensional point set, and then the rotation is determined based on the matching three-dimensional point set matrix and translation matrix, and finally determine the point cloud data of forklift pallets and shelves based on the rotation matrix and translation matrix, achieving the point cloud data of forklift pallets and shelves through the checkerboard labeling method, making the acquisition of point cloud data easier and more accurate , making the subsequent distance difference calculation more accurate, and the accuracy of the forklift's automatic delivery.

In addition, embodiments of the present application also propose a storage medium on which an unmanned forklift high-shelf stocking program is stored. When the unmanned forklift high-shelf stocking program is executed by the processor, the above-mentioned steps are implemented. Steps of unmanned forklift high-level shelf loading method.

Since this storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described again here.

Referring to Figure 5, Figure 5 is a structural block diagram of the first embodiment of the unmanned forklift high shelf loading device of the present application.

As shown in Figure 5, the unmanned forklift high-level shelf placement device proposed by the embodiment of the present application includes:

The image acquisition module 10 is used to obtain image information of the forklift pallets and shelves through the depth camera on the fork arm of the unmanned forklift.

The point cloud extraction module 20 is used to determine the point cloud data of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf.

The position determination module 30 is used to determine the spatial position information of the forklift pallet and the shelf according to the point cloud data.

The difference calculation module 40 is used to determine the distance difference between the forklift pallet and the front surface of the shelf according to the spatial position information.

The adjustment module 50 is used to adjust the position of the fork arm according to the distance difference to complete the placement of goods.

In one embodiment, the point cloud extraction module 20 is further configured to determine the point cloud information of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf; Match with the preset checkerboard pattern to obtain a matching three-dimensional point cloud set; determine the rotation matrix and translation matrix from the depth camera to the unmanned forklift body according to the matching three-dimensional point cloud set; according to the rotation matrix, the The translation matrix and the point cloud information of the forklift pallet and the shelf determine the point cloud data of the forklift pallet and the shelf.

In one embodiment, the point cloud extraction module 20 is further configured to determine the camera coordinate system point cloud data of the forklift pallet and the shelf based on the point cloud information of the forklift pallet and the shelf; The translation matrix converts the point cloud data of the camera coordinate system into the point cloud data of the forklift coordinate system; the point cloud data of the forklift coordinate system is used as the point cloud data of the forklift pallet and the shelf.

In one embodiment, the point cloud extraction module 20 is further configured to determine the grayscale image of the forklift pallet and the shelf based on the point cloud information of the forklift pallet and the shelf; determine based on the grayscale image of the forklift pallet and the shelf. Checkerboard corner points; fit the checkerboard plane formula according to the grayscale image of the forklift pallet and the shelf and the checkerboard corner points; fit the grayscale image of the forklift pallet and the shelf according to the checkerboard plane formula Search and match the points to obtain a two-dimensional point cloud set with the same abscissa and ordinate; determine the homography matrix according to the two-dimensional point cloud set; determine the matching according to the homography matrix and the checkerboard plane formula A collection of 3D point clouds.

In one embodiment, the point cloud extraction module 20 is also used to determine the point cloud image corresponding to the grayscale image of the forklift pallet and the shelf in the forklift coordinate system based on the point cloud data of the forklift coordinate system; The point cloud image corresponding to the grayscale image of the forklift pallet and the shelf in the forklift coordinate system determines the region of interest on the point cloud image corresponding to the grayscale image of the forklift pallet and the shelf in the forklift coordinate system; traverse the region of interest , obtain the point cloud data of forklift pallets and shelves.

In one embodiment, the position determination module 30 is also used to perform discrete point filtering, normal vector filtering, point cloud smoothing, and point cloud clustering on the point cloud data to segment the pallet legs and the shelves. The target point cloud data; use the RANSAC algorithm on the target point cloud data to obtain point cloud plane information; determine the surface point cloud mean data of the forklift pallet and the surface point cloud of the shelf based on the point cloud plane information Data; determining the spatial position information of the forklift pallet and the shelf according to the surface point cloud mean data and the surface point cloud data.

In one embodiment, the adjustment module 50 is also used to compare the distance difference with the posture adjustment threshold to obtain a comparison result; when the comparison result is that the distance difference is greater than the position adjustment threshold, When the posture adjustment threshold is reached, the fork arm is controlled to perform posture adjustment according to the distance difference; when the comparison result is that the distance difference is less than or equal to the posture adjustment threshold, the posture of the fork arm is not adjusted. .

Since this device adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described again here.

It should be understood that the above are only examples and do not constitute any limitation on the technical solution of the present application. In specific applications, those skilled in the art can make settings as needed, and the present application does not impose any limitations on this.

It should be noted that the workflow described above is only illustrative and does not limit the scope of protection of the present application. In practical applications, those skilled in the art can select part or all of it for implementation according to actual needs. The purpose of this embodiment is not limited here.

In addition, for technical details that are not described in detail in this embodiment, please refer to the unmanned forklift high-level shelf loading method provided by any embodiment of this application, and will not be described again here.

Furthermore, it should be noted that, as used herein, the terms "include," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that includes a list of elements includes not only those elements, but also other elements not expressly listed or elements inherent to the process, method, article or system. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

The above serial numbers of the embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or that contributes to the existing technology. The computer software product is stored in a storage medium (such as a read-only memory). , ROM)/RAM, magnetic disk, optical disk), including several instructions to cause a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of this application.

The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of the present application may be directly or indirectly used in other related technical fields. , are all equally included in the patent protection scope of this application.

Claims

[Correction 27.07.2023 under Rule 91]
An unmanned forklift high-level shelf loading method, wherein the unmanned forklift high-level shelf loading method includes:

Obtain image information of forklift pallets and shelves through the depth camera on the fork arm of the unmanned forklift;

Determine the point cloud data of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf;

Determine the spatial position information of the forklift pallet and the shelf according to the point cloud data;

Determine the distance difference between the forklift pallet and the front surface of the shelf based on the spatial position information;

The fork arm is controlled to adjust its posture according to the distance difference to complete the delivery of goods.
[Correction 27.07.2023 under Rule 91]
The method of claim 1, wherein determining the point cloud data of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf includes:

Determine the point cloud information of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf;

Match the point cloud information of the forklift pallets and shelves with the preset checkerboard grid to obtain a matching three-dimensional point cloud set;

Determine the rotation matrix and translation matrix from the depth camera to the unmanned forklift body according to the matching three-dimensional point set;

The point cloud data of the forklift pallet and the shelf are determined according to the rotation matrix, the translation matrix and the point cloud information of the forklift pallet and the shelf.
[Correction 27.07.2023 under Rule 91]
The method of claim 2, wherein determining the point cloud data of the forklift pallet and the shelf based on the rotation matrix, the translation matrix and the point cloud information of the forklift pallet and the shelf includes:

Determine the camera coordinate system point cloud data of the forklift pallet and the shelf based on the point cloud information of the forklift pallet and the shelf;

Convert the camera coordinate system point cloud data into forklift coordinate system point cloud data according to the rotation matrix and the translation matrix;

The point cloud data of the forklift pallet and the shelf are determined according to the point cloud data of the forklift coordinate system.
[Correction 27.07.2023 under Rule 91]
The method of claim 2, wherein matching the point cloud information of the forklift pallets and shelves with a preset checkerboard to obtain a matching three-dimensional point cloud set includes:

Determine the grayscale images of the forklift pallet and the shelf based on the point cloud information of the forklift pallet and the shelf;

Determine the checkerboard corner points based on the grayscale images of the forklift pallets and shelves;

The checkerboard plane formula is fitted based on the grayscale images of the forklift pallets and shelves and the checkerboard corner points;

Search and match points on the grayscale images of the forklift pallets and shelves according to the checkerboard plane formula to obtain a two-dimensional point cloud set with the same abscissa and ordinate;

Determine a homography matrix based on the two-dimensional point cloud collection;

A matching three-dimensional point cloud set is determined according to the homography matrix and the checkerboard plane formula.
[Correction 27.07.2023 under Rule 91]
The method of claim 3, wherein determining the point cloud data of the forklift pallet and the shelf based on the point cloud data of the forklift coordinate system includes:

Determine the point cloud image corresponding to the grayscale image of the forklift pallet and the shelf in the forklift coordinate system based on the forklift coordinate system point cloud data;

Determine the region of interest on the point cloud image corresponding to the grayscale image of the forklift pallet and the shelf in the forklift coordinate system based on the point cloud image corresponding to the grayscale image of the forklift pallet and the shelf in the forklift coordinate system;

Traverse the area of interest to obtain point cloud data of forklift pallets and shelves.
[Correction 27.07.2023 under Rule 91]
The method of claim 1, wherein determining the spatial position information of the forklift pallet and the shelf based on the point cloud data includes:

Perform discrete point filtering, normal vector filtering, point cloud smoothing, and point cloud clustering on the point cloud data to segment the target point cloud data of the pallet legs and the shelf;

Use the RANSAC algorithm on the target point cloud data to obtain point cloud plane information;

Determine the surface point cloud mean data of the forklift pallet and the surface point cloud data of the shelf based on the point cloud plane information;

The spatial position information of the forklift pallet and the shelf is determined based on the surface point cloud mean data and the surface point cloud data.
[Correction 27.07.2023 under Rule 91]
The method according to any one of claims 1 to 6, wherein the controlling the fork arm to perform posture adjustment according to the distance difference to complete the delivery of goods includes:

Compare the distance difference with the pose adjustment threshold to obtain a comparison result;

When the comparison result is that the distance difference is greater than the posture adjustment threshold, control the fork arm to perform posture adjustment according to the distance difference;

When the comparison result is that the distance difference is less than or equal to the posture adjustment threshold, the posture of the fork arm is not adjusted.
[Correction 27.07.2023 under Rule 91]
An unmanned forklift high-level shelf placement device, wherein the unmanned forklift high-level shelf placement device includes:

The image acquisition module is used to obtain image information of forklift pallets and shelves through the depth camera on the fork arm of the unmanned forklift;

A point cloud extraction module is used to determine the point cloud data of the forklift pallet and the shelf based on the image information of the forklift pallet and the shelf;

A position determination module, configured to determine the spatial position information of the forklift pallet and the shelf based on the point cloud data;

A difference calculation module, configured to determine the distance difference between the forklift pallet and the front surface of the shelf based on the spatial position information;

The adjustment module for placing goods is used to control the fork arm to adjust the position and posture according to the distance difference to complete the placement of goods.
[Correction 27.07.2023 under Rule 91]
An unmanned forklift high-level shelf stocking equipment, wherein the equipment includes: a memory, a processor, and an unmanned forklift high-level shelf stocking program stored on the memory and executable on the processor, the The unmanned forklift high-level shelf loading program is configured to implement the unmanned forklift high-level shelf loading method as described in any one of claims 1 to 7.
[Correction 27.07.2023 under Rule 91]
A storage medium, wherein an unmanned forklift high-level shelf stocking program is stored on the storage medium. When the unmanned forklift high-level shelf stocking program is executed by a processor, the unmanned forklift high-level shelf stocking program implements the requirements of any one of claims 1 to 7. The above-mentioned unmanned forklift high-level shelf loading method.