EP4768420A1 - METHOD FOR AUTOMATED FORKHEEL PALLET TRANSFER WITH SIMPLE CAMERA CALIBRATION - Google Patents

Abstract

An enhanced method for automated forklift (10, 20) pallet (90) transfer is disclosed, wherein camera (100, 101) calibration, essential for accurate pallet (90) pose estimation, is performed without external markers. The calibration assumes the camera (100, 101) yaw is initially set to 0 by mechanical means. The camera (100, 101) pitch and roll are calculated from the camera (100, 101)'s image (101). Specifically, each captured image (101) shows parallel forks (20) whose longest outer edges converge at a vanishing point, VP, distinct from the image (101) center IC. Using other pairs of mutually parallel and opposite planes one facing the other, and forks (20)' symmetry, together with the ground plane (1, 2) normal vector, enables the calculation of coordinate transformation T_FC and the inverse coordinate transformation T_CF , where C and F represent the camera (100, 101) and forklift (10, 20) coordinate systems, respectively. This method simplifies camera (100, 101) calibration and facilitates accurate pallet (90) picking.

Description

Technical Field
• A present disclosure reveals an enhanced method for automated forklift pallet transfer with simple camera calibration which renders the core of the disclosure. In general, the disclosure relates to specific image data processing and image analysis directed to single-camera calibration which does not require external calibration markers.
Technical Problem
• With the rapid development of robotics and automation technologies, Automated Guided Vehicles, AGVs, will gradually replace manually operated forklift trucks and soon become an essential component of next-generation industrial transporting solutions.
• Most of the current solutions available on the market use either track-based or laser-guided navigation. While laser-guided navigation boasts precision and natural laser localization does not require changes to the environment, laser sensors struggle to provide detailed descriptions of the environment. That is why vision-based navigation is becoming more prevalent in the industry. Cameras can provide significantly more information about the environment, when paired with technologies like object classification and detection. Standard automated forklift operation strategies, prevalent in the industry today, rely on a network of fixed paths and high positioning repeatability for ensuring efficient pallet pick-ups and deliveries. This approach is appropriate for fully automated environments where machines perform all pick-up and delivery operations. However, it malfunctions in hybrid workflows which include both automated vehicles and humans sharing the same workspace. Humans do not perform tasks with the same repeatability as machines, so pallets placed by humans can be offset by several centimeters and a few degrees from their expected pose. To enable seamless operation of automated forklifts in this scenario, the system should have the ability to adaptively pick up inaccurately placed pallets.
• The essence of reliable pick-up and pallet transport is fast and reliable camera calibration. Keeping in mind that the recent modern vision systems used in AGVs, i.e., automated forklifts, are based on multiple cameras, renders the calibration problem even more important. The essential problem solved with this invention is camera calibration, which is performed without any external markers and is based on a few simple assumptions, making the method extremely fast, i.e. easy to calculate, and reliable.
• The first assumption is that it is always possible to secure, via mechanical means, a single camera on the forklift's frame with the yaw set to 0 degrees for a sufficiently long period. The second assumption is that the camera's pitch and roll are calculable from the camera's image which captured parallel, symmetric forks whose longest outer edges intersect at the vanishing point, VP, which differs from the camera's image center, IC. The method uses other pairs of mutually parallel and opposite planes one facing the other, and forks' symmetry, together with the ground plane normal vector, for the calculation of coordinate transformation T_FC and the inverse coordinate transformation T_CF where C and F stands for camera and forklift coordinate systems respectively.
• Once the calibration method is done, the present method discloses the pick-up and transfer of the pallet with the steps which minimise the number of iterations, during the pick-up process, i.e., during the incorrect pallet pose.
• In one further embodiment, obstacle avoidance is also briefly discussed.
State of the Art
• The state of the art is extremely populated with patent and non-patent literature.
• Korean patent KR101095579B1 for A METHOD FOR POSITIONING AND ORIENTING OF A PALLET BASED ON MONOCULAR VISION, filed in the name of Pusan National University Industry-University Cooperation Foundation, [KR], seems to be the closest prior art for the hereby disclosed method. According to the abstract, this invention relates to a mono vision-based pallet position and attitude measuring method that enables the three-dimensional position and attitude of the pallet in front of the forklift by using a single camera installed in the fork carriage of the forklift. During the camera calibration process, the vanishing point and the forks' geometry are used in the calibration process, but not, for instance, the forks' inner planes which are symmetrical and face one another. The main disadvantage of the cited method is that the type of the pallet should be known for proper functioning, i.e., Korean standard pallet type. In this cited prior art, only the camera yaw is calculated in the same way as proposed by the disclosed invention. All other transformation parameters are deduced differently.
• Chinese patent application CN116309882A for UNMANNED FORKLIFT APPLICATION-ORIENTED TRAY DETECTION AND POSITIONING METHOD AND SYSTEM, is filed in the name of Zhejiang University, [CN]. According to the abstract, the cited invention discloses a tray detection and positioning method - and system for unmanned forklift applications. In the present patent application, the calibration is performed in advance, as is obvious from the description. Namely, the cited step 3.2 reads: calculating three-dimensional coordinates (X, Y, Z) of pixel points of the support column area under a camera coordinate system according to camera parameters calibrated in advance, and via the formula given in the description.
• Chinese patent application CN118644532A for DEPTH VISUAL IDENTIFICATION AND POSITIONING METHOD FOR AGV TERMINAL OPERATION, which is filed in the name of Anhui Heli Co Ltd., [CN]. This patent application is selected because the AGV's path adjustment is performed via Bezier courses, which is rather similar to the technique used in the present disclosure.
• The article Zhengyou Zhang, A FLEXIBLE NEW TECHNIQUE FOR CAMERA CALIBRATION, Microsoft Technical Report MSR-TR-98-71. According to the article's abstract, a flexible new technique to easily calibrate a camera is cited. It is well suited for use without specialized knowledge of 3D geometry or computer vision. Also, the technique only requires the camera to observe a planar pattern shown at a few, at least two, different orientations. Either the camera or the planar pattern can be freely moved. The motion need not be known. Radial lens distortion is modelled. The proposed procedure consists of a closed-form solution, followed by a nonlinear refinement based on the maximum likelihood criterion. This article is selected as an important prior art for CN118644532A main calibration method, and is mentioned in KR101095579B1 as the supportive technique for the camera calibration. Furthermore, it implies the importance of the proper camera calibration.
• There are several articles below which seems to be important as the general prior art.
• Xiao, J., Lu, H., Zhang, L., & Zhang, J. (2017). PALLET RECOGNITION AND LOCALIZATION USING AN RGB-D CAMERA. International Journal of Advanced Robotic Systems, 14(6), 172988141773779. doi:10.1177/1729881417737799. The article reveals the usage of a single low-cost RGB-D camera for solving the technical problem, i.e., to enable a forklift to insert the forks within the pallet's slots for loading and unloading packages. However, the camera calibration is performed manually with the help of the walls to establish transformation relations between the forklift frames and camera frames.
• V. -D. Vu et al., OCCLUSION-ROBUST PALLET POSE ESTIMATION FOR WAREHOUSE AUTOMATION, in IEEE Access, vol. 12, pp. 1927-1942, 2024, doi: 10.1109/ACCESS.2023.3348781. According to section A of the cited document, extrinsic calibration involves placing markers on the checkerboard corners and attaching spherical markers to the sensor. This enabled the estimation of the transformation between the motion capture system's pose and the RGB-D camera's optical frame in the cited document.
• Zhao, J.; Li, B.; Wei, X.; Lu, H.; Lü, E.; Zhou, X. RECOGNITION AND LOCATION ALGORITHM FOR PALLETS IN WAREHOUSES USING RGB-D SENSOR. Appl. Sci. 2022, 12, 10331. https://doi.org/10.3390/app122010331. This document represents yet another general state-of-the-art document, solving the problem of large uncertainties when using an RGB-D sensor to recognize and locate pallets in warehouse environments. It seems that the document is silent regarding the camera calibration at all.
• None of the above-cited documents reveals such a quick and reliable method for the camera calibration process, which can be easily executed on the standard AGV's hardware, with desired accuracy.
Summary of the Invention
• A method for automated forklift pallet transfer within a warehouse is disclosed.
• An automated forklift comprises means for receiving, storing, and processing digital information, where the information is either pre-stored or loaded from external sources, and acquired from sensors including at least one indoor or outdoor positioning system, and a depth camera attached to the forks' frame below or above the plane where forks are positioned. In addition, the forklift has the means for autonomous driving, equipped with safety mechanisms for recognizing obstacles.
• The disclosed method is represented as a task list executed as a computer program stored in the forklift's processing unit. The method comprises the following steps A-G mentioned below.
1. A. Loading a warehouse roadmap in the world frame, where the roadmap consists of interconnected nodes, with each node representing points of interest in a warehouse, including intersections, forklift charging positions, pre-pallet nodes, pallet nodes, and node connector curves (NCC), together with the conditions for following the NCCs.
2. B. Loading the next task in the task list that the forklift must execute, wherein each task specifies the goal node, task type, and associated task parameters which define what actions the forklift has to perform.
3. C. Path planning using the data from step A from the current forklift pose to the goal node defined in step B, where the calculated path is the one that satisfies imposed conditions for each NCC, defined in step A.
4. D. Depending on the task type defined in step B, the vehicle executes different actions along the path calculated in step C.
5. E. All actions that require following the path defined in step C use a control method selected from Model Predictive Control, Linear quadratic control, Pure pursuit control, or Gain scheduling control, implemented for autonomous driving of the forklift, utilizing localization on the roadmap by the implemented positioning system, for driving between the nodes of interest while following the node connector curves, and optionally using the program to avoid the obstacles.
6. F. If the action in step E requires pickup of the pallet, the forklift follows the path to the pre-pallet node of the path, the method performs safety checks for pallet pickup and possible corrections, the forklift enters the pallet node of the path to pick up the pallet and finally exit back to the pre-pallet node.
7. G. Finally, the said method is restarted from step B.
• The said method has the following sub-steps within step F. Once the forklift approaches the specified pallet for pick-up and transfer, at the pre-pallet node the method executes the following sub-steps:
1. i) runs the camera calibration procedure to calculate the coordinate transformation T_FC from the forklift frame F to the camera frame C and the corresponding inverse coordinate transformation T_CF,
2. ii) lifts forks to pallet height,
3. iii) estimates the pallet pose using the RGB-D camera, where the taken RGB image undergoes a semantic segmentation, and where the segmented image and the corresponding depth image are used to calculate the pallet pose in the camera frame,
4. iv) calculates the pallet pose in the world frame, using the known forklift pose in the world frame as a reference, with the transformations T_FC and T_CF from step i) and pose from step iii) in the camera frame, and
5. v) if the pallet is misaligned over the predetermined threshold when comparing the expected pallet pose in the roadmap with the estimated pallet pose from step iv), the forklift exits the current pre-pallet node to a previously passed node, a new pre-pallet node is calculated based on pallet pose in step iv) and a new path is planned from the current forklift position to the new pre-pallet node, the roadmap is temporarily updated for a new path and new pre-pallet node, which the forklift approaches via the said newly calculated path, and
6. vi) the steps iii) - v) are repeated until the pallet is not detected as misaligned in step v), at which point the pallet is picked up by the forklift, and the instructions of step F are finalized.
• The present method is characterized by its camera calibration in step i), which is performed without any external markers, based on the assumption that the camera yaw is set initially to 0 degrees by mechanical means, while the camera pitch and camera roll are calculated from the camera's image. This image always shows parallel, symmetric forks whose longest outer edges intersect at the vanishing point, VP, which differs from the camera's image center, IC. Using other pairs of mutually parallel and opposite planes one facing the other, and forks' symmetry, together with the ground plane normal vector, allows the calculation of coordinate transformation T_FC and the inverse coordinate transformation T_CF.
• The method also discloses exact calculation steps for the calculation of the forklift frame F origin (x_F ,y_F ,z_F ) and the camera yaw, roll, and pitch (ϕ,ψ,θ) parameters.
• In the preferred embodiment, the RGB-D camera is positioned above the plane where forks are positioned, measured from the ground. Furthermore, it is favourable that the camera is positioned on the fork frame so that at least 30 cm of the forks are visible from the camera image. Also, the node connectors curves in the method are preferably smooth continuous curves, such as Non-uniform rational B-spline (NURBS) curves. The obstacle avoidance program is programmed to calculate the obstacle go-around path with at least two new intersections and one node connector added to the initially calculated path in the method's step C. The indoor positioning system is preferably selected to be a 2D or 3D lidar.
• The mentioned forklift is designed to comprise means for carrying out the above-cited method.
Description of Figures
• Figure 1 depicts the AGV, i.e., the forklift and the pallet in the scene's side view.
• Figure 2 shows the same scene as Figure 1, from the bird perspective.
• Figure 3 shows a part of the forklift with the forklift's frame F and the camera's frame C origins and the corresponding axes' orientation.
• Figure 4 shows a part of the forklift with the forklift's frame F and the auxiliary camera frame C' origins and the corresponding axes' orientation.
• Figure 5 is the picture taken by RGB-D camera, with an indicated vanishing point, VP, image center, IC, and all relevant geometries for performing the camera calibration method.
Detailed Description of the Invention
• In this section, the method is discussed up to the details sufficient for repeating the invention by the person skilled in the art. In addition, some technical setups, that are needed to perform the invention, are briefly addressed.
Forklift in general and other hardware parts
• A forklift (10), with the up-down movable frame (40) to which is attached the left (20) and right (30) fork is as schematically depicted in Figures 1 and 2. Figures show the forklift (10) which is driving over the ground plane (1), surrounded by the walls (2) that intersects at the warehouse's horizon line (3). Forklift (10) is in task to pick-up the pallet (90), with or without load (91), as depicted in Figure 5.
• Regarding the term "forklift", in this disclosure this term has a broader meaning and includes any pallet transportation means where the present camera calibration method can be applied. Having in mind that, for simplicity, the forks (20, 30) are defined as essential for carrying out the method described herewith. Also, the said forks (20, 30) are designed to perform the main tasks - to be inserted into the pallets' pockets (92, 93). But, the person skilled in the art will immediately recognize that instead of the mentioned forks (20, 30), any vehicle's parallel edges that may appear under the camera (100) sight, and which may serve to establish the vanishing point, VP, may be equally used for the camera calibration process disclosed herewith, if additional faced planes are visible on the taken image (101).
• So, the present method and technology are easy to apply to all sorts of similar vehicles, such as underride robots, unit load carrier robots, etc, without any inventive effort. The list of such vehicles can be found at: https://go4robotics.com/amr-and-agv-types/, THE MOST COMMON AMR AND AGV, published by IFR - International Federation of Robotics, non-profit organization, 2024.
• Every such automated forklift, defined in a wider sense has, besides the standard means for transferring pallets including propulsion means, driving system, and lifting means, some specific means needed for autonomous driving, which include safety mechanisms for recognizing obstacles as well. Furthermore, such automated forklift comprises means for receiving, storing, and processing digital information, where the information is either pre-stored or loaded from external sources, and acquired from sensors including at least one indoor or outdoor positioning system, and a depth sensor and image sensor attached to the forks frame (40) below or above the plane where forks are positioned. In most cases, one RGB-D camera (100) is sufficient to operate with the said method as a depth sensor and image sensor which renders the method inexpensive for practical use.
• The person skilled in the art will recognize the possibility of emulating RGB-D camera (100) with a standard RGB camera and 2D or 3D lidar, or stereo camera to establish the same technical effect, i.e., to capture the RGB image and simultaneously capture or calculate the depth field needed in some instances of carrying out the proposed method.
Method for automated forklift pallet transfer
• The disclosed method is set as a task list, or list of instructions, that is executed as a computer program stored in the forklift's processing unit. The method comprises the following steps A-G, explained below.
• Step A. The method begins with loading a warehouse roadmap which is prepared in the world frame. The forklift estimates its position using one or more indoor or outdoor positioning systems built for this purpose. The roadmap consists of interconnected nodes. Each node represents a point of interest in a warehouse. Such nodes represent but are not limited to intersections, forklift charging positions, pre-pallet nodes, pallet nodes, and node connector curves - NCC, together with the conditions for following the NCCs. The specific conditions may include various constraints, such as speed limits, steering speeds, and other constraints along the NCC, which are well described in EP4073608B1 , A METHOD FOR ACCURATE AND EFFICIENT CONTROL OF AUTOMATED GUIDED VEHICLES FOR LOAD TRANSPORTATION TASKS, filed in the name of RoMb Technology d.o.o. This step is in principle executed once and repeated only in case of recorded changes in the warehouse, for instance, the layout of the warehouse is changed or pallet racks are moved, etc.
• Step B. This step begins with the loading of the first or any following, i.e., the "next task" from the stored task list that the forklift must execute. Each task specifies the goal node, task type, and associated task parameters which define what actions the forklift has to perform during the task.
• Step C. The forklift path is planned by using the data from step A from the current forklift pose, which includes the forklift's orientation and the position on the roadmap, i.e., starting node, to the goal node defined in step B. The calculated path is the one that satisfies imposed conditions for each NCC, which are defined in step A. The algorithm used for path planning can be any algorithm based on a minimum cost function or a similar heuristic algorithm, which is suitable for the use case.
• Step D. Depending on the task type defined in step B, the vehicle executes different actions along the path calculated in step C.
• Step E. All actions that require following the path defined in step C use an appropriate control method. Such control method can be selected from Model Predictive Control, Linear quadratic control, Pure pursuit control, or Gain scheduling control, but it is not limited to them. The selected method is implemented for autonomous forklift driving, utilizing localization on the roadmap by the implemented positioning system, for driving between the nodes of interest while following the node connector curves. Optionally, it is possible to use the program to avoid the obstacles. The person skilled in the art will understand that any suitable method can be used as the control method instead of those mentioned above. Similarly, for the obstacle avoidance process, any suitable method in the art can be implemented.
• Step F. If the action in step E requires pickup of the pallet, the forklift follows the path to the pre-pallet node of the path. Then, the method performs safety checks for pallet pickup and possible corrections, the forklift enters the pallet node of the calculated path to pick up the pallet and finally exit back to the pre-pallet node.
• Step G. Finally, the said method is restarted from step B.
• The said method has the following sub-steps within step F above, which refines the procedure once the forklift approaches the specified pallet for pick-up and transfer, at the pre-pallet node. The method performs sub-steps i)-vii):
1. i) runs the camera calibration procedure to calculate the coordinate transformation T_FC from the forklift frame F to the camera frame C and the corresponding inverse coordinate transformation T_CF,
2. ii) lifts forks to pallet height,
3. iii) estimates the pallet pose using the RGB-D camera, where the taken RGB image undergoes a semantic segmentation, and where the segmented image and the corresponding depth image are used to calculate the pallet pose in the camera frame,
4. iv) calculates the pallet pose in the world frame, using the known forklift pose in the world frame as a reference, with the transformations T_FC and T_CF from step i) and pose from step iii) in the camera frame,
5. v) if the pallet is misaligned over the predetermined threshold when comparing the expected pallet pose in the roadmap with the estimated pallet pose from step iv), the forklift exits the current pre-pallet node to a previously passed node, a new pre-pallet node is calculated based on pallet pose in step iv) and a new path is planned from the current forklift position to the new pre-pallet node, the roadmap is temporarily updated for a new path and new pre-pallet node, which the forklift approaches via the said newly calculated path, and
6. vi) the steps iii) - v) are repeated until the pallet is not detected as misaligned in step v), at which point the pallet is picked up by the forklift, and the instructions of step F are completed.
• The above method is described via main steps A-G, and sub-steps i)-vi) of step F. The important and non-trivial of this method is situated in the sub-step i). The camera calibration method in step i) is performed without any external markers, and based on a few assumptions.
• The first assumption is that the camera yaw is set initially to 0 degrees by mechanical means. This can be done by using a simple level and a few fastening screws for fastening the camera (100) to the frame (40) in a way that the camera top is horizontal to the ground, by which the rotation over Z-axes in the camera frames C and C' is blocked. For frames, please check Figures 3 and 4.
• The second assumption is that the camera pitch and roll are calculable from the camera's image (101), Figure 5, which shows parallel, symmetric forks whose longest outer edges intersect at the vanishing point, VP, which differs from the camera's image centre, IC. So, using other pairs of mutually parallel and opposite planes one facing the other, and forks' symmetry, together with the ground plane normal vector, allows the calculation of coordinate transformation T_FC and the inverse coordinate transformation T_CF.
Camera calibration procedure
• It is well-known in the art that T_FC and T_CF transformations are defined via rotational and translation matrices R_ij, t_ij respectively, where i,j relates to the selected frames F,C and auxiliary frame C' applied on a position vector x = [x y z] ^T to read: $${\mathbf{x}}_F=T_{\mathit{FC}}\left({\mathbf{x}}_C\right)=R_{\mathit{FC}}{\mathbf{x}}_C+t_{\mathit{FC}}$$
• Vector x_C = [x y z] ^T contains 3-D coordinates in the original coordinate camera frame C, and vector x_F = [x' y' z'] ^T contains 3-D coordinates in the forklift frame F. The T_FC transformation can be written as a series of transformations among the frames: C → C' → F, or in the present notation: $$T_{\mathit{FC}}=T_{\mathit{FC}\prime }\ast T_{C\prime C}$$
• The auxiliary nominal camera frame C' is defined via transformation: $$T_{\mathit{FC}\prime }\left(\mathbf{x}\right)=R_{\mathit{FC}\prime }\left(\mathbf{x}\right)+t_{\mathit{FC}\prime }$$ where $$R_{\mathit{FC}\prime }=Z_{\frac{\pi }{2}}{Y}_0{X}_{-\frac{\pi }{2}}$$ and $$T_{C\prime C}\left(\mathbf{x}\right)=R_{C\prime C}\left(\mathbf{x}\right)+t_{C\prime C}$$
• Matrix R_C'C describes the rotation around the camera's nominal axis defining the ideal camera position, Figures 3 and 4, as: $$R_{C\prime C}=X_{\psi }{Y}_{\theta }{Z}_{\varphi }$$
• Matrices X_ψ, Y_θ, Z_ϕ denote elementary rotation operators around X, Y, and Z axes fixed to the nominal camera frame by angles ψ, θ and ϕ respectively and with the translation vector set to null vector: $$t_{C\prime C}={\left[\begin{array}{ccc}0& 0& 0\end{array}\right]}^T$$
• It is worth looking at Figure 3 which shows the relation among the most general camera frame C and forklift frame F, where O_C and O_F are origins of the said frames. The purpose of a newly introduced auxiliary or ideal O_C , frame is evident from Figure 4, where the relation among these two frames, expressed as rotations ( $\psi =-\frac{\pi }{2}$ , θ = 0, $\varphi =\frac{\pi }{2}$ ) is evident, while separated by the translation vector t_FC'.
• The following steps S1-S4 are dedicated to the calculation of the forklift frame's origin O_F, expressed as (x_F ,y_F ,z_F ) in the camera frame C. Furthermore, the camera's yaw, roll, and pitch (ϕ,ψ,θ) calculations are disclosed below as well.
• S1: The camera yaw setting is defined by the angle ϕ. It is set to 0, which is physically achieved by mechanical means as explained before, by affixing the camera to a horizontal position via vehicle structure.
• S2: For camera roll calculation, defined with the angle ψ, the following procedure is provided. The mounted camera (100) sees the forks and the ground below the said forks. The camera roll is calculated by calculating the angle between the ground plane, Gp, and the camera's x-z plane which is the angle between the ground plane's normal vector n _Gp and the camera negative y-axis vector y _C = [0 -1 0] ^T where: $$\psi =-\arccos \left|\frac{{\mathbf{n}}_{\mathit{Gp}}\cdot {\mathbf{y}}_C}{\left|{\mathbf{n}}_{\mathit{Gp}}\right|\cdot \left|{\mathbf{y}}_C\right|}\right|$$
• The said ground plane normal vector n _GP, see Figure 5, is calculated by the Random Sample Consensus, RANSAC, algorithm, which is a common algorithm in image processing. The RANSAC algorithm is taken over the camera point cloud, which represents the drivable terrain. The drivable terrain is detected by semantic segmentation of the RGB image. The output of the RANSAC algorithm is the normal of the plane with the most inliers.
• S3: Now it is possible to perform the camera pitch calculation, defined with the angle θ. The procedure uses VP and IC points, from the taken image (101), as visible in Figure 5. The following equation is used: $$\theta =\mathit{arctan}\frac{{\mathit{VP}}_x-{\mathit{IC}}_x}{f_x}$$
• Index x refers to the x-coordinate measured in camera pixels, see Figure 5, for the VP and IC points, and focal length f expressed in pixels. Having in mind that the pixels are not necessarily rectangular, the f_x value specifies the focal length expressed in pixels.
• After obtaining camera yaw, roll, and pitch (ϕ,ψ,θ) values, the translation vector t_FC, must be calculated. This is performed in step S4 described below.
• S4: Calculation of the translation vector t_C'F among the frames F, C' must be performed, by calculating x_F, y_F and z_F coordinate of the forklift frame origin expressed in O_C'. The translation vector t_C'F is: $$t_{C\prime F}={\left[x_F{y}_F{z}_F\right]}^T$$
• For the x_F coordinate calculation of the forklift frame F origin O_F, the normal vectors n _P2 and n _P3 of the planes P ₂ and P ₃ are used. P ₂ and P ₃ planes, see Figure 5, approximate the right side of the left fork and the left side of the right fork. The RANSAC algorithm is used again, and it returns the plane coefficients for P ₂ and P ₃ planes and the corresponding inliers. Looking at Figures 4 and 5, it is easy to establish: $$x_F=\frac{\left|P_2{P}_3\right|}{2}-\left|O_{C\prime }{P}_3\right|$$ where $$\left|P_2{P}_3\right|=\left|O_{C\prime }{P}_2\right|+\left|O_{C\prime }{P}_3\right|$$
• Value |P ₂ P ₃| represents the distance between the said planes and |O_C'P ₂| and |O_C'P ₃| are distances of the said planes from the camera origin O_C' = [0 0 0] ^T in C' frame. Therefore, x_F measures the deviation of the camera with respect to the said forks' planes.
• Coordinate y_F is obtained by calculating the distance of C' origin O_C' from the ground plane Gp using step S2 data for Gp plane. This calculation is performed via a point-to-plane distance formula, knowing Gp plane data, so we have: $$y_F=\left|O_{C\prime }\mathit{Gp}\right|$$
• In addition, z_F coordinate of the forklift frame origin is obtained via the following steps: $${\mathbf{n}}_{P5}={\mathbf{n}}_{\mathit{Gp}}\times {\mathbf{n}}_{P2}$$
• Vector n _P5 is the normal vector of the plane P ₅ containing the fork tip oriented towards the pallet, see Figure 5. In addition, this plane has the maximum value of the Z-coordinate in the point cloud, which represents forks points, obtained using semantic segmentation. The data obtained in steps S2 and S3 contain n _Gp and n _P2 values. Value z_F is, therefore: $$z_F=\left|O_{C\prime }{P}_5\right|-{\mathit{OFFS}}_z$$ where OFFS_z is the offset between F frame origin to the end of the forks, which position z_F among the forks (20, 30).
• The translation vector is finally calculated via the expression: $$t_{\mathit{FC}\prime }=-R_{C\prime F}^T{t}_{C\prime F}$$ by which all transformations among F,C and auxiliary frame C' are uniquely defined.
Practical use of the method
• The above-cited method can be carried out with the RGB-D camera positioned above or below the said forks. However, practical use gives some advantages to the setup where the said camera is positioned above the plane where forks lie, measured from the ground. In addition, the experimental data and simulations suggest that the camera should be mounted on the fork frame so that at least 30 cm of the forks are visible from the camera image, for good performance.
• Considering the main method part, especially method step C, the method's node connectors' curves are preferably smooth continuous curves, more preferably, a Non-uniform rational B-spline, NURBS, curves that produce optimum AVG performance in driving. Similarly, in step E, the obstacle avoidance program, if engaged, calculates the obstacle go-around path with at least two new intersections and one node connector added to the initially calculated path in step C, again applying the NURBS curves for go-around paths.
• At the present state of the art in the field of indoor and outdoor positioning, 2D or 3D lidar has an unprecedented characteristic that cannot be simply substituted with another positioning system. An example of such a system is the Decawave DW1000 chip functioning in Ultra-Wide-Band. Namely, this technique has a significant position error of approximately +10 cm and problematic spreading among the metal framed shelfs due to the multiple reflections.
• Finally, the person skilled in the art will, without making specific assumptions, use an adequate automated forklift, and the RGB-D camera to perform the method, or, the RGB camera and appropriate lidar, to perform the method described herewith.
Industrial Applicability
• The industrial applicability of the said disclosure is obvious. A method for automated forklift pallet transfer is disclosed, wherein camera calibration, essential for accurate pallet pose estimation, is performed without external markers.
• The disclosed calibration method assumes that the camera yaw is initially set to 0 by mechanical means, while the camera pitch and roll are calculated from the camera's image. Specifically, each captured image shows parallel forks whose longest outer edges converge at a vanishing point, VP, distinct from the image center IC. Using other pairs of mutually parallel and opposite planes one facing the other, and forks' symmetry, together with the ground plane normal vector, enables the calculation of coordinate transformation T_FC and the inverse coordinate transformation T_CF, where C and F represent the camera and forklift coordinate systems, respectively. This method simplifies camera calibration and facilitates accurate pallet picking.
Reference numbers

1

Ground plane

2

Walls

3

Horizon line

10

Forklift

20

Left Fork

30

Right Fork

40

Movable frame

90

Pallete

91

Load

92

Pallete pocket

93

Pallete pocket

100

Camera

101

Image

Claims (8)

1. A method for automated forklift pallet transfer within a warehouse, wherein the forklift comprises:

   - means for receiving, storing, and processing digital information, where the information is either pre-stored or loaded from external sources, and acquired from sensors including at least one indoor or outdoor positioning system, and a depth sensor and image sensor, such as RGB-D camera, attached to the forks' frame below or above the plane where forks are positioned

   - means for autonomous driving, equipped with safety mechanisms for recognizing obstacles,

   where the method is a task list that is executed as a computer program stored in the forklift's processing unit, and the method comprises the following steps:

   A. loading a warehouse roadmap in the world frame, where the roadmap consists of interconnected nodes, with each node representing points of interest in a warehouse, including intersections, forklift charging positions, pre-pallet nodes, pallet nodes, and node connector curves, NCC, together with the conditions for following the NCCs,

   B. loading the next task in the task list that the forklift must execute, wherein each task specifies the goal node, task type, and associated task parameters which define what actions the forklift has to perform,

   C. path planning using the data from step A from the current forklift pose to the goal node defined in step B, where the calculated path is the one that satisfies imposed conditions for each NCC, defined in step A,

   D. depending on the task type defined in step B, the vehicle executes different actions along the path calculated in step C,

   E. all actions that require following the path defined in step C use a control method selected from Model Predictive Control, Linear quadratic control, Pure pursuit control, or Gain scheduling control, implemented for autonomous driving of the forklift, utilizing localization on the roadmap by the implemented positioning system, for driving between the nodes of interest while following the node connector curves, and optionally using the program to avoid the obstacles, and

   F. if the action in step E requires pickup of the pallet, the forklift follows the path to the pre-pallet node of the path, the method performs safety checks for pallet pickup and possible corrections, the forklift enters the pallet node of the path to pick up the pallet and finally exit back to the pre-pallet node, and

   G. the said method is restarted from step B;

   where the said method has the following sub-steps within step F, once the forklift approaches the specified pallet for pick-up and transfer at the pre-pallet node, where the method:

   i) runs the camera calibration procedure to calculate the coordinate transformation T_FC from the forklift frame F to the camera frame C and the corresponding inverse coordinate transformation T_CF,

   ii) lifts forks to pallet height,

   iii) estimates the pallet pose using the RGB-D camera, where the taken RGB image undergoes a semantic segmentation, and where the segmented image and the corresponding depth image are used to calculate the pallet pose in the camera frame,

   iv) calculates the pallet pose in the world frame, using the known forklift pose in the world frame as a reference, with the transformations T_FC and T_CF, from step i) and pose from step iii) in the camera frame, and

   v) if the pallet is misaligned over the predetermined threshold when comparing the expected pallet pose in the roadmap with the estimated pallet pose from step iv), the forklift exits the current pre-pallet node to a previously passed node, a new pre-pallet node is calculated based on pallet pose in step iv) and a new path is planned from the current forklift position to the new pre-pallet node, the roadmap is temporarily updated for a new path and new pre-pallet node, which the forklift approaches via the said newly calculated path,

   vi) the steps iii) - v) are repeated until the pallet is not detected as misaligned in step v), at which point the pallet is picked up by the forklift, and the instructions of step F are finalized;

   characterized in that camera calibration in step i) is performed without any external markers, based on the assumption that the camera yaw is set initially to 0 degrees by mechanical means, while the camera pitch and camera roll are calculated from the camera's image which shows parallel, symmetric forks whose longest outer edges intersect at the vanishing point, VP, which differs from the camera's image center, IC, and using other pairs of mutually parallel and opposite planes one facing the other, and forks' symmetry, together with the ground plane normal vector, allows the calculation of coordinate transformation T_FC and the inverse coordinate transformation T_CF.
2. The method according to claim 1, where T_FC and T_CF transformations are defined via rotational and translation matrices R_ij, t_ij respectively, where i,j relates to the selected frames F,C and auxiliary frame C' applied on a position vector x = [x y z] ^T to read: $${\mathbf{x}}_F=T_{\mathit{FC}}\left({\mathbf{x}}_C\right)=R_{\mathit{FC}}{\mathbf{x}}_C+t_{\mathit{FC}}$$

   where vector x_C= [x y z] ^T contains 3-D coordinates in the original camera coordinate frame C, and vector x_F = [x' y' z'] ^T contains 3-D coordinates in the forklift frame F, and where T_FC transformation is written: $$T_{\mathit{FC}}=T_{\mathit{FC}\prime }\ast T_{C\prime C}$$

   where the auxiliary nominal camera frame C' is defined via transformation: $$T_{\mathit{FC}\prime }\left(\mathbf{x}\right)=R_{\mathit{FC}\prime }\left(\mathbf{x}\right)+t_{\mathit{FC}\prime }$$

   where $$R_{\mathit{FC}\prime }=Z_{\frac{\pi }{2}}{Y}_0{X}_{-\frac{\pi }{2}}$$ and $$T_{C\prime C}\left(\mathbf{x}\right)=R_{C\prime C}\left(\mathbf{x}\right)+t_{C\prime C}$$

   where R_C'C describes the rotation around the camera's nominal axis defining the ideal camera position as: $$R_{C\prime C}=X_{\psi }{Y}_{\theta }{Z}_{\varphi }$$

   with X_ψ, Y_θ, Z_ϕ denoting elementary rotation operators around X, Y, and Z axes fixed to the nominal camera frame by angles ψ, θ and ϕ respectively and with the translation vector set to null vector: $$t_{C\prime C}={\left[\begin{array}{ccc}0& 0& 0\end{array}\right]}^T$$

   characterized by the following calculation steps S1-S4 for the calculation of the forklift frame F origin (x_F,y_F,z_F ) and the camera yaw, roll, and pitch (ϕ,ψ,θ) values,

   S1: the camera yaw, defined by the angle ϕ, is set to 0, which is achieved by mechanical means, affixing the camera to a vehicle structure that holds the camera horizontally,

   S2: for camera roll calculation, defined with the angle ψ, the mounted camera sees the forks and the ground below the said forks, and the camera roll is calculated by calculating the angle between the ground plane, Gp , and the camera's x-z plane which is the angle between the ground plane's normal vector n _Gp and the camera negative y-axis vector y _C = [0 -1 0] ^T where: $$\psi =-\arccos \left|\frac{{\mathbf{n}}_{\mathit{Gp}}\cdot {\mathbf{y}}_C}{\left|{\mathbf{n}}_{\mathit{Gp}}\right|\cdot \left|{\mathbf{y}}_C\right|}\right|$$ and the said ground plane n _GP is calculated by the Random Sample Consensus, RANSAC, an algorithm using points from the camera-taken point cloud which represents the drivable terrain; where the drivable terrain is detected by semantic segmentation of the RGB image, and where the output of the RANSAC algorithm is the normal of the plane which has the most inliers,

   S3: for camera pitch calculation, defined with the angle θ, the procedure uses VP and IC points from the taken image, where: $$\theta =\mathit{arctan}\frac{{\mathit{VP}}_x-{\mathit{IC}}_x}{f_x}$$

   S4: calculation of the translation vector t_C'F among the frames F, C' by calculating x_F, y_F and z_F coordinate of the forklift frame origin: $$t_{C\prime F}={\left[x_F{y}_F{z}_F\right]}^T$$

   where for the x_F coordinate calculation of the forklift frame F origin, the normal vectors n _P2 and n _P3 of the planes P ₂ and P ₃ are used, where P ₂ and P ₃ planes approximate the right side of the left fork and the left side of the right fork which are faced one to another, and where the RANSAC algorithm returns the plane coefficients for P ₂ and P ₃ planes and the corresponding inliers, where: $$x_F=\frac{\left|P_2{P}_3\right|}{2}-\left|O_C,P_3\right|$$

   and where: $$\left|P_2{P}_3\right|=\left|O_C,P_2\right|+\left|O_C,P_3\right|$$

   while |P ₂ P ₃| represents the distance between the said planes, while |O_C'P ₂| and |O_C'P ₂| represents the distance of the said planes P ₂, P ₃ from O_C' = [0 0 0] ^T ,

   where y_F is obtained by calculating the distance of C' origin O_C, from the ground plane Gp using step S2 data for Gp plane and formula for point-to-plane distance: $$y_F=\left|O_{C\prime }\mathit{Gp}\right|$$

   where z_F coordinate of the forklift frame origin is obtained via the following steps: $${\mathbf{n}}_{P5}={\mathbf{n}}_{\mathit{Gp}}\times {\mathbf{n}}_{P2}$$

   where n _P5 is the normal vector of the plane P ₅ containing the fork tip oriented towards the pallet with the maximum value of the Z-coordinate in the point cloud, which represents forks points, obtained using semantic segmentation, while data obtained in steps S2 and S3 contain n _Gp and n _P2 values, where $$z_F=\left|O_{C\prime }{P}_5\right|-{\mathit{OFFS}}_z$$

   and OFFS_z is the distance between F frame origin to the end of the forks, and the translation vector: $$t_{\mathit{FC}\prime }=-R_{C\prime F}^T{t}_{C\prime F}$$

   by which all transformations among F,C and auxiliary frame C' are uniquely defined.
3. The method according to claim 1 or 2, wherein the RGB-D camera is positioned above the plane where forks are positioned, measured from the ground.
4. The method according to claim 3, wherein the camera is positioned on the fork frame so that at least 30 cm of the forks are visible from the camera image.
5. The method according to claim 1, where node connectors curves are smooth continuous curves, preferably, Non-uniform rational B-spline, NURBS, curves.
6. The method according to claim 1, where the obstacle avoidance program calculates the obstacle go-around path with at least two new intersections and one node connector added to the initially calculated path in step C.
7. The method according to any of the preceding claims, where the indoor positioning system is preferably a 2D or 3D lidar.
8. A forklift comprising means for carrying out the method of claims 1-7.