CN114170521B - Forklift pallet butt joint identification positioning method - Google Patents
Forklift pallet butt joint identification positioning method Download PDFInfo
- Publication number
- CN114170521B CN114170521B CN202210130296.3A CN202210130296A CN114170521B CN 114170521 B CN114170521 B CN 114170521B CN 202210130296 A CN202210130296 A CN 202210130296A CN 114170521 B CN114170521 B CN 114170521B
- Authority
- CN
- China
- Prior art keywords
- tray
- information
- forklift
- neural network
- deep learning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a forklift pallet butt joint identification and positioning method, which comprises the following steps: acquiring image information of current forklift butt joint, wherein the image information comprises an RGB (red, green and blue) image and a depth image; inputting the RGB image into a pre-established and trained deep learning neural network, and judging whether the deep learning neural network outputs the identified tray area information; when the effective tray area information is output and recognized by the deep learning neural network, acquiring coordinates of each tray leg and pixel information of a tray area under a camera coordinate system based on the recognized tray area information; and judging whether the distance between adjacent tray legs meets a preset condition or not based on the coordinates of each tray leg, if so, acquiring the 3D point cloud of the tray area based on the pixel information of the tray area, matching the 3D point cloud with the pre-established tray template point cloud, and acquiring the position information of the forklift relative to the tray. The method can effectively reduce the mismatching probability caused by the false detection of the tray and simultaneously reduce the calculation complexity.
Description
Technical Field
The invention relates to the technical field of robots, in particular to a forklift pallet butt joint identification and positioning method.
Background
The unmanned forklift is an important transportation carrier in production automation and logistics automation systems, and has the functions of automatic navigation and walking, automatic forking of tray goods feeding, automatic stacking and the like. The automatic forking tray needs to accurately sense the tray and calculate the position relation of the tray relative to the fork arms, and then the fork arms are automatically guided to stretch into the tray holes in the middle through navigation control, and the fork arms are lifted to fork the tray after the tray is moved in place.
At present, a single line laser radar is often used in the prior art to identify a tray and calculate a position relation of the tray relative to a forklift, specifically, the single line laser radar is used to scan obstacle information near a tray placement area behind a fork arm, a laser 2D contour map can be obtained, the obtained 2D contour map is matched with a tray 2D point cloud template with known specification and size (for example, a template matching method such as ICP is adopted), the position of the tray under a current camera coordinate system can be obtained, then the position of the fork arm relative to the tray can be calculated, and therefore the forklift is automatically guided to complete a tray butt joint process through navigation motion control.
However, the above mentioned single line lidar docking method has the following disadvantages:
the first and the second 2D laser radars can only obtain the information of the 2D plane, that is, the positioning is performed by scanning the three tray foot information of the tray, the point clouds of the three tray feet are not much, and the point clouds are easily interfered by other similar objects around, and the matching cannot be guaranteed all the time.
Secondly, because the tray is generally high only about 10CM, for scanning the tray leg, laser radar scanning plane need install and be less than 10CM position, and because laser radar installation levelness, reasons such as ground flatness under this condition, laser radar scans ground very easily, and then leads to butt joint matching failure. And the laser radar is arranged at the tip of the fork arm, so that the thickness of the fork arm has certain mechanical structure requirements, and the laser radar does not have installation conditions under many conditions.
In addition, those skilled in the art have developed the use of depth cameras to identify pallets in order to overcome the deficiencies of lidar. Specifically, the depth image information or the 3D point cloud information of the tray can be obtained by shooting through the depth camera, then the position of the tray under a current camera coordinate system is obtained by calculating through 3D point cloud matching or other methods, compared with a 2D laser radar, the depth camera can obtain dense depth image information (information of a three-dimensional space), the information quantity is richer than that of the 2D laser radar, and the robustness and the precision of detection can be improved. However, no matter the 2D laser radar or the depth camera is used, since the obtained information is geometric information such as point cloud, if there is an interfering object similar to a tray leg in the environment, there is a possibility of a docking failure due to misrecognition, and meanwhile, in an actual operation site, it is often necessary to fork a plurality of trays of different specifications, and a matching process needs to try tray templates of different specifications and sizes, which further increases the probability of matching the interfering object in the environment, and may cause an operation accident.
Disclosure of Invention
Technical problem to be solved
In view of the above disadvantages and shortcomings of the prior art, the present invention provides a method for identifying and positioning pallet butt joint of a forklift.
(II) technical scheme
In order to achieve the purpose, the invention adopts the main technical scheme that:
in a first aspect, an embodiment of the present invention provides a forklift pallet docking identification and positioning method, including:
s10, acquiring image information of current forklift butt joint in the process of carrying out pallet butt joint by the forklift, wherein the image information comprises an RGB (red, green and blue) image and a depth image;
s20, inputting the RGB image into a pre-established and trained deep learning neural network, and judging whether the deep learning neural network outputs the identified tray area information;
s30, when the depth learning neural network outputs the identified tray area information, acquiring coordinates of each tray leg and pixel information of the tray area in a camera coordinate system based on the identified tray area information and depth information corresponding to the tray area information in the depth image;
s40, judging whether the distance between adjacent tray legs meets a preset condition or not based on the coordinates of each tray leg, if so, acquiring 3D point cloud of the tray area based on the pixel information of the tray area, and performing point cloud matching with a pre-established tray template to acquire the position information of the forklift relative to the tray;
the deep learning neural network is pre-established and used for identifying a tray area in an input RGB image and outputting identified tray area information, wherein the tray area information comprises: pixel location of the tray area and center pixel location of each tray leg in the RGB image.
Optionally, before the S10, the method further includes:
building a deep learning neural network for identifying the tray area, wherein the input of the deep learning neural network is an RGB image, and the output is the pixel position of the tray area in the RGB image and the central pixel position of each tray leg;
obtaining training data, the training data comprising: the method comprises the steps that images including all trays in different angles, different distances and/or different background environments are obtained, and the peripheral outline of the tray and the central pixel position information of each tray foot are marked in each image in advance;
and training the deep learning neural network by adopting gradient back propagation based on the training data to obtain the trained deep learning neural network with adaptive weight parameters (namely, the trained weight parameters are the selected optimal weight parameters, and the output result of the deep learning neural network adopting the optimal weight parameters is more accurate).
Optionally, the deep learning neural network comprises: a 10-tier network structure;
the 1 st, 2 nd, 4 th, 5 th, 7 th and 8 th layers are convolution layers, the 3 rd, 6 th and 9 th layers are pooling layers or down-sampling layers, the last layer is an output layer, and the pixel position of the output tray area is determined;
alternatively, the first and second electrodes may be,
the deep learning neural network comprises: a 10-tier network structure;
the 1 st, 2 nd, 4 th, 5 th, 7 th and 8 th layers are convolution layers, the 3 rd, 6 th and 9 th layers are pooling layers or down-sampling layers, the last layer is an output layer, and the pixel positions of the tray area represented by a rectangular frame are output.
Optionally, the S30 includes:
finding depth information corresponding to each central pixel in the depth image based on the central pixel of each tray foot in the tray area information, and calculating the coordinate of each central pixel in a camera coordinate system;
specifically, assuming that the central pixel value of the first pallet leg is (u, v) and the corresponding depth value in the depth image is d, the coordinate value of the central pixel value in the camera coordinate system (x, y, z) is obtained based on formula one as follows:
wherein, f is the focal length of the RGBD camera for acquiring the RGB image, cx is the optical center pixel value of the RGBD camera in the horizontal direction, and cy is the optical center pixel value of the RGBD camera in the vertical direction.
Optionally, the S40 includes:
if the coordinates of each tray foot in the camera coordinate system are: c1 = (x1, y1, z1), C2 = (x2, y2, z2), C3 = (x3, y3, z3), then the euclidean space distance d1 between the coordinate C1 and the coordinate C2, and the euclidean space distance d1 between the coordinate C2 and the coordinate C3 are calculated2；
The Euclidean space distance d1Distance dt from pallet legs of known pallet formwork1Comparing if it is less than a predetermined thresholddthAnd the Euclidean space is separated by a distance d2Distance dt from pallet legs of known pallet formwork2Comparing;
namely, the preset conditions are as follows:
and if the preset conditions are met, determining that the distance between the adjacent tray legs meets the preset conditions.
Optionally, the S40 includes:
based on the pixel position of the tray area in the tray area information, searching the depth information of the tray area in the depth image, and acquiring coordinate values (x, y, z) of all pixel points of the tray area in a camera coordinate system;
and generating a 3D point cloud of the tray area based on the coordinate values (x, y, z) of all pixel points of the tray area under the camera coordinate system, matching the 3D point cloud with a pre-established tray template point cloud, and acquiring the position information of the forklift relative to the tray.
Optionally, matching the 3D point cloud with a pre-established tray template point cloud in an iterative closest point manner.
Optionally, the method further comprises:
if the deep learning neural network does not output the identified tray area information, the tray is not in the camera view field or is blocked by an obstacle (if all or part of the tray is blocked by the obstacle), and the identification is finished;
and if the distance between the adjacent tray legs does not accord with the preset condition, finishing the identification.
In a second aspect, the invention also provides a forklift, wherein an RGBD camera is installed on the rear of the forklift body, the RGBD camera is electrically connected with a forklift control device, and the control device executes the forklift tray butt joint identification and positioning method according to any one of the first aspect.
In a third aspect, an embodiment of the present invention further provides a control device for a forklift, including: the device comprises a memory and a processor, wherein the memory is used for storing a computer program, and the processor is used for executing the computer program stored in the memory and executing the steps of the forklift pallet docking identification and positioning method in any one of the first aspect.
(III) advantageous effects
The method of the embodiment of the invention uses the deep learning neural network to detect/identify the tray area on the color image (namely RGB image), and the use of the semantic level information enables the detection result to be more stable and reliable, thereby greatly reducing the problem of mismatching of the traditional 2D laser radar or depth camera method.
The method also adopts a deep learning neural network to simultaneously detect the tray leg information, compares the tray leg information with the known tray specification size, and ensures the reliability of the tray detection as a check.
According to the method, the tray area is detected by using the deep learning neural network, and the point cloud of the area is generated in the depth map for matching and positioning calculation, so that the calculation complexity is greatly reduced compared with the traditional method of matching by using all point cloud data, and the calculation speed can be effectively increased.
Drawings
Fig. 1 is a schematic flow chart of a forklift pallet docking, identifying and positioning method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of a forklift pallet docking identification and positioning method according to another embodiment of the present invention;
FIG. 3 is a schematic illustration of the relative positions of the forks, cargo, RGBD camera and pallet in a forklift;
fig. 4 is a schematic diagram of information output by the deep learning neural network.
Detailed Description
For the purpose of better explaining the present invention and to facilitate understanding, the present invention will be described in detail by way of specific embodiments with reference to the accompanying drawings.
The embodiment of the invention provides a deep learning-based RGBD camera forklift pallet butt joint recognition positioning method, which is divided into an off-line training stage and an on-line recognition stage in specific processing, wherein in the off-line training stage, a pallet detection deep learning neural network is firstly built, then a plurality of color pictures (different angles and different position relations) of pallets to be butted are shot, after marking, training samples are generated, and then the neural network is trained; when on-line identification is carried out, firstly, color pictures shot by an RGBD camera are input into a depth learning neural network for calculation, rectangular areas occupied by trays in the pictures and central point pixel information of tray legs are output, if no tray is detected or the detected position relation between the tray legs is not consistent with the actual specification and size of the tray, the current identification is judged to be failed, and the calculation is finished, otherwise, depth data of each pixel in the rectangular areas of the trays are intercepted on a depth map shot by the RGBD camera, 3D point clouds are generated, and the 3D point clouds are matched with a preset tray template for calculation to obtain the relative position between the tray and the RGBD camera.
Example one
As shown in fig. 1, an embodiment of the present invention provides a method for identifying and positioning pallet butt joint of a forklift, where an execution subject of the method of this embodiment may be a control device of the forklift, and the following steps are all steps corresponding to an online identification stage in the second embodiment; the RGBD camera is installed on the forklift of the embodiment, the RGBD camera is installed behind the forklift body, and is installed horizontally or slightly obliquely downwards, so that the rear area of the fork arm is located in the visual field of the RGBD camera, as shown in FIG. 3.
The forklift pallet docking, identifying and positioning method of the embodiment can comprise the following steps:
s10, acquiring image information of current forklift docking in the process of forklift tray docking, wherein the image information comprises RGB images and depth images.
In this embodiment, the RGBD camera may perform acquisition according to a preset acquisition frame rate, for example, the preset acquisition frame rate may be 5 Hz. The RGBD camera can obtain a color image (such as an RGB image) and a depth map (i.e., a depth image) from each shooting, and each pixel position of the color image and the depth map corresponds to each other. The color image is a common camera picture, and the depth image stores the actual distance of each pixel from the camera.
S20, inputting the RGB image into a pre-established and trained deep learning neural network, and judging whether the deep learning neural network outputs the identified tray area information; if so, step S30 is executed, otherwise, it can be stated that the tray is not in the field of view of the camera, or the tray is blocked, the online identification of the method is ended, and the method of this embodiment is executed again based on the new image information captured by the RGBD camera.
S30, when the deep learning neural network outputs the identified tray region information, acquiring coordinates of each tray leg and pixel information of the tray region in the camera coordinate system based on the identified tray region information and depth information corresponding to the tray region information in the depth image, as shown in fig. 4;
and S40, judging whether the distance between adjacent tray legs meets a preset condition or not based on the coordinates of each tray leg, if so, acquiring the 3D point cloud of the tray area based on the pixel information of the tray area, and performing point cloud matching with a pre-established tray template to acquire the position information of the forklift relative to the tray.
Of course, if the preset condition is not met, it may be determined that the online recognition of the method is finished, and the method of the present embodiment is executed based on new image information captured by the RGBD camera.
It should be noted that, in this embodiment, the deep learning neural network is a pre-established network for identifying a tray area in an input RGB image and outputting tray area information of the identification, where the tray area information includes: pixel position of the tray area and center pixel position of each tray leg in the RGB image.
In the embodiment, the deep learning neural network is used for detecting/identifying the tray area on the color image (namely, the RGB image), the detection result is more stable and reliable by means of semantic level information, and the problem of mismatching of the traditional 2D laser radar or depth camera method can be greatly reduced.
In addition, the deep learning neural network is adopted to simultaneously detect the tray leg information, and compared with the known tray specification size, the tray leg information is used as a check to ensure the reliability of tray detection, namely, the mismatching probability caused by the misdetection of the tray is effectively reduced.
According to the method, the tray area is detected by using the deep learning neural network, and then the point cloud of the area is generated in the depth map for matching and positioning calculation, compared with the traditional method of matching by using all point cloud data, the calculation complexity is greatly reduced, and the calculation speed can be effectively improved.
Example two
The method for identifying and positioning the pallet butt joint of the forklift according to the embodiment of the invention is described in detail with reference to fig. 2 and 3 to 4. In the embodiment, each shooting of the RGBD camera can obtain a color image and a depth map, and each pixel position of the color image and the depth map corresponds to each other. The color map is a common camera picture, and the depth map stores the actual distance of each pixel from the camera.
The method of the present embodiment can be divided into two stages, i.e., an off-line training stage and an on-line recognition stage, which are described in detail below.
An off-line training stage:
the first step is as follows: and constructing a deep learning neural network for detecting the tray, wherein the input of the deep learning neural network is a color image, the output is the pixel area position of the tray in the color image, the pixel area position is represented by a rectangular frame, and meanwhile, the output layer also outputs the central pixel position of each tray leg in the color image.
For example, the deep learning neural network in this embodiment includes a 10-layer network structure, the 1 st, 2 nd, 4 th, 5 th, 7 th, 8 th layers may be convolutional layers, the 3 rd, 6 th, 9 th layers may be pooling layers or down-sampling layers, the last layer is an output layer, the output is a pixel region position where the tray is located in the color picture, which is generally represented by a rectangular frame, and the output layer also outputs the central pixel position of each tray leg in the color picture. The output of the deep learning neural network in this embodiment may be represented in the form of other boxes, and this embodiment is not limited to a rectangular box and is set according to actual needs.
The second step: training weight parameters of the deep learning neural network; and training the weight parameters of the built deep learning neural network. The method comprises the steps of shooting a plurality of color pictures of trays to be butted in advance by using an RGBD camera, marking the peripheral outline of the trays and the central pixel position of each tray foot in each color picture in the plurality of color pictures by using a known marking tool to serve as training data (ground channel), and training the deep learning neural network weight by using gradient back propagation to obtain a better deep learning neural network weight.
It should be noted that the more data the picture is taken, the better the picture is, and the different angles, different distances and/or different background environments from the tray are taken, so that the deep learning neural network training effect can be improved, and the output result is more accurate in use. Of course, the training data may be obtained in various ways, for example, the actual conditions of material on the tray, no material on the tray, inclined tray, other tray on the side, equipment, legs, etc. may also be photographed, and the peripheral outline of the tray and the central pixel position/central pixel value of each tray foot are marked in these color photographs by using a marking tool in advance.
And (3) an online identification stage:
in the process of butting the pallet by the forklift, calling the following calculation steps at a certain frequency so as to calculate the position of the pallet relative to the forklift at each moment in real time, wherein the method comprises the following steps of:
s.101, shooting a frame of picture by using an RGBD camera, wherein the frame of picture comprises a color image to be analyzed and a depth image to be analyzed;
s.102, inputting the color image into the trained deep learning neural network, and judging whether the deep learning neural network outputs the identified tray area;
if the deep learning neural network does not output the rectangular tray area, the tray is not detected, the tray is not in the camera visual field or is blocked, and the online identification is finished; otherwise, go to step s.103.
S.103, central pixel values of all tray feet output by the trained deep learning neural network are obtained, and the depth of the corresponding pixel is found in the depth map.
For example, assuming that the pixel value of a certain tray leg is (u, v) and the depth value of the corresponding pixel in the depth map is d, the coordinate value of the position (x, y, z) of the center pixel in the camera coordinate system can be calculated as:
wherein f is the focal length of the RGBD camera acquiring the RGB image, cx is the optical center pixel value of the RGBD camera in the horizontal direction, cy is the optical center pixel value of the RGBD camera in the vertical direction, the coordinate values of all the detected tray leg center pixels can be obtained according to the above calculation method, and are marked as C1 = (x1, y1, z1), C2 = (x2, y2, z2), C3 = (x3, y3, z3), and the euclidean spatial distances d between C1 and C2, between C2 and C3 are calculated respectively1，d2Distance dt from the pallet legs of the known pallet template1，dt2The comparison is carried out, if less than a predetermined threshold value dth, that is to say
The preset conditions are as follows:
and (5) only if the preset conditions are met, the position relation of the tray legs output by the tray detection deep learning neural network is consistent with the actual tray, the step S.104 is carried out, and otherwise, the tray identification is finished.
S.104, finding out a rectangular tray area output by the tray detection depth learning neural network in the depth map, calculating by the same method as the step S.103 to obtain x, y and z of all pixel points in the area, thereby generating a 3D point cloud of the area, carrying out point cloud matching (for example, carrying out point cloud matching by adopting a known ICP (inductively coupled plasma) on the point cloud and the known tray template point cloud, namely calculating to obtain the position of the camera relative to the tray, and finishing the tray identification and positioning.
In the embodiment, the deep learning neural network is adopted to simultaneously detect the tray leg information, and the tray leg information is compared with the known tray specification size to be used as verification, so that the reliability of tray detection is ensured.
Particularly, the deep learning neural network is used for detecting the tray area and then generating the point cloud of the area in the depth map for matching and positioning calculation, compared with the traditional method of matching by using all point cloud data, the calculation complexity is greatly reduced, and the calculation speed can be effectively improved.
EXAMPLE III
The embodiment also provides a forklift, wherein an RGBD camera is mounted on the rear of a forklift body of the forklift, as shown in fig. 3, the RGBD camera is electrically connected with a control device of the forklift, and the control device executes the forklift tray butt joint identification and positioning method in any one of the first embodiment and the second embodiment.
In addition, this embodiment still provides a fork truck's controlling means, includes: a memory and a processor; the processor is used for executing the computer program stored in the memory to realize the steps of executing the forklift pallet docking identification and positioning method in any of the first embodiment and the second embodiment.
It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the terms first, second, third and the like are for convenience only and do not denote any order. These words are to be understood as part of the name of the component.
Furthermore, it should be noted that in the description of the present specification, the description of the term "one embodiment", "some embodiments", "examples", "specific examples" or "some examples", etc., means that a specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, the claims should be construed to include preferred embodiments and all changes and modifications that fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include such modifications and variations.
Claims (6)
1. A forklift tray butt joint identification and positioning method is characterized in that an RGBD camera is mounted on a forklift and mounted behind a forklift body of the forklift so that a rear area of a fork arm is located in the visual field of the RGBD camera, and comprises the following steps:
building a deep learning neural network for identifying the tray area, wherein the input of the deep learning neural network is an RGB image, and the output is the pixel position of the tray area in the RGB image and the central pixel position of each tray leg;
obtaining training data, the training data comprising: the method comprises the following steps that images including all trays in different angles, different distances and/or different background environments are obtained, and the peripheral outline of each tray and the central pixel position information of each tray foot are marked in each image in advance;
training a deep learning neural network by adopting gradient back propagation based on the training data to obtain the trained deep learning neural network with adaptive weight parameters;
s10, in the process of carrying out tray butt joint by the forklift, a control device of the forklift acquires image information of current forklift butt joint, wherein the image information comprises an RGB image and a depth image; the RGBD camera collects image information according to a 5Hz collection frame rate;
s20, inputting the RGB image into a pre-established and trained deep learning neural network, and judging whether the deep learning neural network outputs the identified tray area information;
s30, when the depth learning neural network outputs the identified tray area information, acquiring coordinates of each tray leg and pixel information of the tray area in a camera coordinate system based on the identified tray area information and depth information corresponding to the tray area information in the depth image;
finding depth information corresponding to each central pixel in the depth image based on the central pixel of each tray foot in the tray area information, and calculating the coordinate of each central pixel in a camera coordinate system;
specifically, assuming that the central pixel value of the first pallet leg is (u, v) and the corresponding depth value in the depth image is d, the coordinate value of the central pixel value in the camera coordinate system (x, y, z) is obtained based on formula one as follows:
wherein f is the focal length of an RGBD camera for acquiring an RGB image, cx is an optical center pixel value of the RGBD camera in the horizontal direction, and cy is an optical center pixel value of the RGBD camera in the vertical direction;
s40, judging whether the Euclidean space distance between adjacent tray legs meets a preset condition or not based on the coordinates of each tray leg, if so, acquiring 3D point cloud of the tray area based on the pixel information of the tray area, and performing point cloud matching with the pre-established tray template point cloud by adopting an ICP (inductively coupled plasma) mode to acquire the position information of the forklift relative to the tray;
the S40 includes: if the coordinates of each tray foot in the camera coordinate system are: c1 = (x1, y1, z1), C2 = (x2, y2, z2), C3 = (x3, y3, z3), then the euclidean space distance d1 between the coordinate C1 and the coordinate C2, and the euclidean space distance d1 between the coordinate C2 and the coordinate C3 are calculated2；
Will Euclidean space distance d1Distance dt from pallet legs of known pallet formwork1Comparing if it is less than a predetermined thresholddthAnd the Euclidean space is separated by a distance d2Distance dt from pallet legs of known pallet formwork2Comparing;
namely, the preset conditions are as follows:
if the preset conditions are met, determining that the distance between adjacent tray legs meets the preset conditions, namely, the position relation of the tray legs output by the deep learning neural network meets the actual tray;
the deep learning neural network is pre-established and used for identifying a tray area in an input RGB image and outputting identified tray area information, wherein the tray area information comprises: pixel location of the tray area and center pixel location of each tray leg in the RGB image.
2. The method of claim 1,
the deep learning neural network comprises: a 10-tier network structure;
the 1 st, 2 nd, 4 th, 5 th, 7 th and 8 th layers are convolution layers, the 3 rd, 6 th and 9 th layers are pooling layers or down-sampling layers, the last layer is an output layer, and the pixel position of the output tray area is obtained;
alternatively, the first and second electrodes may be,
the deep learning neural network comprises: a 10-tier network structure;
the 1 st, 2 nd, 4 th, 5 th, 7 th and 8 th layers are convolution layers, the 3 rd, 6 th and 9 th layers are pooling layers or down-sampling layers, the last layer is an output layer, and the pixel positions of the tray area represented by a rectangular frame are output.
3. The method according to claim 1, wherein the S40 includes:
based on the pixel position of the tray area in the tray area information, searching the depth information of the tray area in the depth image, and acquiring coordinate values (x, y, z) of all pixel points of the tray area in a camera coordinate system;
and generating a 3D point cloud of the tray area based on the coordinate values (x, y, z) of all pixel points of the tray area under the camera coordinate system, matching the 3D point cloud with a pre-established tray template point cloud, and acquiring the position information of the forklift relative to the tray.
4. The method of claim 3,
and matching the 3D point cloud with a pre-established tray template point cloud by adopting an iteration closest point mode.
5. The method of claim 1, further comprising:
if the deep learning neural network does not output the identified tray area information, the tray is not in the camera view field or is blocked by an obstacle, and the identification is finished;
and if the distance between the adjacent tray legs does not accord with the preset condition, finishing the identification.
6. A forklift is characterized in that an RGBD camera is installed on the rear of a forklift body, the RGBD camera is electrically connected with a control device of the forklift, and the control device executes the forklift tray butt joint identification positioning method as claimed in any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210130296.3A CN114170521B (en) | 2022-02-11 | 2022-02-11 | Forklift pallet butt joint identification positioning method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210130296.3A CN114170521B (en) | 2022-02-11 | 2022-02-11 | Forklift pallet butt joint identification positioning method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114170521A CN114170521A (en) | 2022-03-11 |
| CN114170521B true CN114170521B (en) | 2022-06-17 |
Family
ID=80489809
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210130296.3A Active CN114170521B (en) | 2022-02-11 | 2022-02-11 | Forklift pallet butt joint identification positioning method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114170521B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115436964B (en) * | 2022-11-09 | 2023-03-24 | 深圳市壹站智汇科技有限公司 | Object recognition and path planning method, carrier, electronic device, and storage medium |
| CN116310622A (en) * | 2022-12-15 | 2023-06-23 | 珠海创智科技有限公司 | Method and system for accurately identifying tray based on deep learning |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160073094A1 (en) * | 2014-09-05 | 2016-03-10 | Microsoft Corporation | Depth map enhancement |
| CN108171748B (en) * | 2018-01-23 | 2021-12-07 | 哈工大机器人（合肥）国际创新研究院 | Visual identification and positioning method for intelligent robot grabbing application |
| CN111368852A (en) * | 2018-12-26 | 2020-07-03 | 沈阳新松机器人自动化股份有限公司 | Article identification and pre-sorting system and method based on deep learning and robot |
| WO2021068799A1 (en) * | 2019-10-07 | 2021-04-15 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Occlusion and collision detection for augmented reality applications |
| CN111080693A (en) * | 2019-11-22 | 2020-04-28 | 天津大学 | Robot autonomous classification grabbing method based on YOLOv3 |
| CN112001972A (en) * | 2020-09-25 | 2020-11-27 | 劢微机器人科技(深圳)有限公司 | Tray pose positioning method, device and equipment and storage medium |
| CN113408584B (en) * | 2021-05-19 | 2022-07-26 | 成都理工大学 | RGB-D multi-modal feature fusion 3D target detection method |
| CN113537096B (en) * | 2021-07-21 | 2023-08-15 | 常熟理工学院 | AGV forklift warehouse position tray identification and auxiliary positioning method and system based on ROS |
-
2022
- 2022-02-11 CN CN202210130296.3A patent/CN114170521B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN114170521A (en) | 2022-03-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220036574A1 (en) | System and method for obstacle avoidance | |
| JP6348093B2 (en) | Image processing apparatus and method for detecting image of detection object from input data | |
| CN114170521B (en) | Forklift pallet butt joint identification positioning method | |
| EP3168812B1 (en) | System and method for scoring clutter for use in 3d point cloud matching in a vision system | |
| Kang et al. | Automatic targetless camera–lidar calibration by aligning edge with gaussian mixture model | |
| JP5671281B2 (en) | Position / orientation measuring apparatus, control method and program for position / orientation measuring apparatus | |
| US9576363B2 (en) | Object picking system, object detecting device, object detecting method | |
| KR20160003776A (en) | Posture estimation method and robot | |
| US20210039257A1 (en) | Workpiece picking device and workpiece picking method | |
| US20220189055A1 (en) | Item detection device, item detection method, and industrial vehicle | |
| CN109341692B (en) | Line navigation method and robot | |
| CN112070759A (en) | Forklift pallet detection and positioning method and system | |
| JP2016179534A (en) | Information processor, information processing method, program | |
| CN112184793B (en) | Depth data processing method and device and readable storage medium | |
| CN116494253B (en) | Target object grabbing pose acquisition method and robot grabbing system | |
| CN117115784A (en) | Vehicle detection method and device for target data fusion | |
| JPH07103715A (en) | Method and apparatus for recognizing three-dimensional position and attitude based on visual sense | |
| JP2007253699A (en) | Optical axis deviation sensing device | |
| JP7448035B2 (en) | Information processing device, three-dimensional position estimation method, and program | |
| CN114396875A (en) | Rectangular parcel volume measurement method based on vertical shooting of depth camera | |
| CN115249324A (en) | Method and device for determining position to be stacked in stack shape and computing equipment | |
| US20150063637A1 (en) | Image recognition method and robot | |
| CN116309882A (en) | Tray detection and positioning method and system for unmanned forklift application | |
| Varga et al. | Improved autonomous load handling with stereo cameras | |
| CN115797446A (en) | Shelf positioning method, shelf docking method, device, equipment, and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |