CN116468787A - Position information extraction method and device of forklift pallet and domain controller - Google Patents
Position information extraction method and device of forklift pallet and domain controller Download PDFInfo
- Publication number
- CN116468787A CN116468787A CN202310165250.XA CN202310165250A CN116468787A CN 116468787 A CN116468787 A CN 116468787A CN 202310165250 A CN202310165250 A CN 202310165250A CN 116468787 A CN116468787 A CN 116468787A
- Authority
- CN
- China
- Prior art keywords
- forklift pallet
- forklift
- jack
- pallet
- key points
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000605 extraction Methods 0.000 title claims description 6
- 238000000034 method Methods 0.000 claims abstract description 44
- 238000012545 processing Methods 0.000 claims description 36
- 238000001514 detection method Methods 0.000 claims description 26
- 230000006870 function Effects 0.000 claims description 25
- 238000004891 communication Methods 0.000 claims description 18
- 238000004364 calculation method Methods 0.000 claims description 11
- 230000008447 perception Effects 0.000 claims description 9
- 238000012549 training Methods 0.000 claims description 9
- 238000003860 storage Methods 0.000 claims description 6
- 230000015654 memory Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 5
- 238000003384 imaging method Methods 0.000 description 5
- 238000006243 chemical reaction Methods 0.000 description 4
- 230000007547 defect Effects 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000007613 environmental effect Effects 0.000 description 3
- 238000002372 labelling Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000013486 operation strategy Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Forklifts And Lifting Vehicles (AREA)
Abstract
The invention discloses a method and a device for extracting position information of a forklift pallet and a domain controller, wherein the method comprises the following steps: shooting a forklift pallet to be positioned, and acquiring a forklift pallet image; identifying key points near the forklift pallet jacks in the forklift pallet image; and reasoning the space position of the forklift pallet according to the identified key points and the camera internal parameters, so as to obtain the position information used for representing the space position of the forklift pallet, wherein the camera internal parameters are the camera internal parameters for shooting the images of the forklift pallet. The technical scheme provided by the invention can improve the accuracy of positioning the forklift tray by the automatic driving forklift.
Description
Technical Field
The invention relates to the field of automatic driving, in particular to a method and a device for extracting position information of a forklift pallet and a domain controller.
Background
Image-based computer vision is an important module for autopilot. The automatic fork truck realizes the automatic handling of material etc. and has important meaning to unmanned operation, and wherein important one ring is the tray location, only accurate location tray position just can make fork truck travel to the position of settlement to in the middle of inserting the tray with the fork accuracy. In a standardized production process, the trays have certain specifications, namely, a plurality of trays follow a set of fixed length and width external parameters. In order to obtain 3D information, according to the current technical scheme, a tray is positioned based on point cloud matching of a laser radar, but the accuracy of the method is affected by the density of the point cloud and noise, classification information cannot be directly obtained from the point cloud, the laser radar is high in cost and difficult to maintain, waiting time exists based on accumulated point cloud of the solid-state laser radar, and user experience is poor. Another technical scheme is based on the depth coordinate of the direct measurement tray of depth camera, but the depth camera receives natural light's influence very easily, can't have steady operation under outdoor or indoor condition that there is strong light to shine, and range finding distance and precision are limited. Therefore, a forklift pallet position information extraction method with higher accuracy is needed.
Disclosure of Invention
In view of the above, the embodiment of the invention provides a method and a device for extracting position information of a forklift pallet and a domain controller, so that the accuracy of positioning the forklift pallet is improved.
According to a first aspect, an embodiment of the present invention provides a method for extracting position information of a forklift pallet, where the method includes: shooting a forklift pallet to be positioned, and acquiring a forklift pallet image; identifying key points near forklift pallet jacks in the forklift pallet image; and reasoning the space position of the forklift pallet according to the identified key points and the camera internal parameters to obtain the position information used for representing the space position of the forklift pallet, wherein the camera internal parameters are camera internal parameters for shooting the forklift pallet image.
Optionally, the forklift pallet jack is rectangular, identifying a key point near the forklift pallet jack in the forklift pallet image includes: detecting a forklift pallet target from the forklift pallet image; and identifying four vertexes corresponding to each forklift pallet jack from the forklift pallet target through a preset key point detection model, and taking the identified vertexes as key points near the forklift pallet jacks.
Optionally, the keypoint detection model is trained by a modified loss function, wherein the modified loss function is:
L total ＝L 1 +L 2
wherein L is total Representing an improved overall loss function value, L 1 Represents the existing loss function value, L 2 Represents a loss function value calculated based on key points near the forklift pallet jack, n represents the batch number of training samples, i represents the ith batch of training samples, K 1 [1]And K 2 [1]Respectively representing the x-axis coordinate of a first vertex and the x-axis coordinate of a second vertex positioned on the first vertical side of the current forklift pallet jack, K 3 [1]And K 4 [1]Respectively representing the x-axis coordinates of the third vertex and the fourth vertex located on the second vertical side of the current forklift pallet jack.
Optionally, the reasoning is performed on the spatial position of the forklift pallet according to the identified key points and the camera internal parameters to obtain the position information for representing the spatial position of the forklift pallet, including: calculating space coordinates corresponding to each forklift pallet jack by utilizing key points corresponding to each forklift pallet jack and the camera internal parameters; and determining the position information according to the space coordinates corresponding to the jack of each forklift pallet.
Optionally, the calculating the spatial coordinates corresponding to each forklift pallet jack by using the key points corresponding to each forklift pallet jack and the camera internal parameters includes: calculating an average value of xy coordinates of key points of the current forklift pallet jack in the image to obtain xy coordinates of a center point of the current forklift pallet jack; measuring the physical distance between key points of the current forklift pallet jacks from the forklift pallet entity; based on a similar triangle principle, calculating the depth coordinate of the central point of the current forklift pallet jack by utilizing the xy coordinates of the camera internal parameters, the key points of the current forklift pallet jack in the image and the physical distance between the key points of the current forklift pallet jack; and determining the space coordinates of the current forklift pallet jack by utilizing the depth coordinates of the central point of the current forklift pallet jack and the xy coordinates of the central point of the current forklift pallet jack.
Optionally, based on the principle of similar triangle, the calculating the depth coordinate of the center point of the current forklift pallet jack by using the xy coordinates of the camera internal parameter and the key point of the current forklift pallet jack in the image and the physical distance between the key points of the current forklift pallet jack includes: calculating the image distance between the key points of the current forklift pallet jacks by using xy coordinates of the key points of the current forklift pallet jacks in the image; according to the corresponding relation between each image distance and the entity distance, carrying out similar triangle calculation by combining the focal length parameters in the camera internal parameters to obtain a plurality of depth information; calculating the average value of the depth information, and taking the obtained average value of the depth as the actual depth of the center point of the current forklift pallet jack; and determining the depth coordinate of the center point of the current forklift pallet jack by utilizing the actual depth of the center point.
Optionally, the determining the position information according to the space coordinates corresponding to the jack of each forklift pallet includes: extracting two pieces of lateral depth information with different positions from the depth information in the lateral side direction of the forklift pallet; determining the transverse physical distance of the key point corresponding to the transverse depth information in the transverse edge direction of the forklift pallet; calculating the offset angle of the forklift pallet by utilizing the inverse trigonometric function relation between the transverse depth information and the transverse entity distance; and taking the offset angle of the forklift pallet and the space coordinates corresponding to the jacks of each forklift pallet as the position information.
According to a second aspect, an embodiment of the present invention provides a position information extraction device of a forklift pallet, where the device includes: the image acquisition module is used for shooting a forklift pallet to be positioned and acquiring forklift pallet images; the key point identification module is used for identifying key points near the forklift pallet jacks in the forklift pallet image; the tray positioning module is used for reasoning the space position of the forklift tray according to the identified key points and the camera internal parameters, so as to obtain the position information used for representing the space position of the forklift tray, and the camera internal parameters are the camera internal parameters for shooting the forklift tray image.
According to a third aspect, an embodiment of the present invention provides a domain controller, including: the system comprises a perception processing unit, a decision processing unit, a control processing unit and a communication unit, wherein the perception processing unit, the decision processing unit, the control processing unit and the communication unit are in communication connection, computer instructions are stored in the perception processing unit, and the perception processing unit executes the computer instructions so as to execute the method in the first aspect or any optional implementation manner of the first aspect.
According to a fourth aspect, embodiments of the present invention provide a computer readable storage medium storing computer instructions for causing a computer to perform the method of the first aspect, or any one of the alternative embodiments of the first aspect.
The technical scheme that this application provided has following advantage:
according to the technical scheme, the forklift truck tray to be positioned is shot by the forklift truck, and the forklift truck tray image is obtained; identifying key points near the forklift pallet jacks in the forklift pallet image; and reasoning the space position of the forklift pallet according to the identified key points and the camera internal parameters to obtain the position information used for representing the space position of the forklift pallet. Therefore, the scheme provided by the embodiment of the invention completes the 3D positioning of the tray position based on the camera equipment, reduces the use and maintenance cost compared with the laser radar technical scheme, and avoids the defects of limited distance and influence of natural light compared with the depth camera technical scheme. The scheme only needs to train a key point detection model, has small calculated amount, high running speed, simpler realization and easier popularization and application.
Drawings
The features and advantages of the present invention will be more clearly understood by reference to the accompanying drawings, which are illustrative and should not be construed as limiting the invention in any way, in which:
fig. 1 is a schematic step diagram of a method for extracting position information of a forklift pallet according to an embodiment of the present invention;
fig. 2 is a flow chart illustrating a method for extracting position information of a forklift pallet according to an embodiment of the present invention;
FIG. 3 illustrates a schematic view of a forklift pallet image taken in one embodiment of the present invention;
FIG. 4 illustrates a camera imaging schematic in one embodiment of the invention;
FIG. 5 shows a schematic representation of an image sample annotated in one embodiment of the invention;
fig. 6 is a schematic structural diagram of a position information extraction device of a forklift pallet according to an embodiment of the present invention;
fig. 7 is a schematic diagram showing the structure of a domain controller according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, based on the embodiments of the invention, which a person skilled in the art would obtain without making any inventive effort, are within the scope of the invention.
Referring to fig. 1 and 2, in one embodiment, a method for extracting position information of a forklift pallet specifically includes the following steps:
step S101: shooting a forklift pallet to be positioned, and acquiring a forklift pallet image.
Step S102: and identifying key points near the forklift pallet jacks in the forklift pallet image.
Specifically, the embodiment of the invention is to install a high-resolution camera for a forklift and is used for shooting images, and as shown in fig. 3, the high-resolution camera is a forklift tray image containing forklift trays. Then, key points near the forklift pallet jacks are identified from the forklift pallet images by a pre-trained key point detection model. Taking the rectangular forklift pallet jack in fig. 3 as an example, the jack is used for inserting the fork of the forklift so as to enable the forklift to drag up the pallet, and transport the goods on the pallet, in practical application, the shape of the actual forklift pallet jack is standard, for example, oval, circular, and the like. The key points detected by the embodiment of the invention are that a plurality of points capable of representing the outline of the forklift pallet jack are identified from the periphery of the forklift pallet jack.
Step S103: and reasoning the space position of the forklift pallet according to the identified key points and the camera internal parameters, so as to obtain the position information used for representing the space position of the forklift pallet, wherein the camera internal parameters are the camera internal parameters for shooting the images of the forklift pallet.
Specifically, as shown in fig. 4, a schematic diagram of the camera capturing an image P' of the object P is shown. The camera internal parameters include a focal length of the camera, an optical center, a camera coordinate system, an imaging plane of the camera, etc., and the key points identified in step S102 have a certain geometric relationship on the imaging plane of the camera, and similarly, the identified key points are corresponding to the forklift pallet entity, and the actual geometric relationship between the key points is measurable (for example, the actual distance between the key points and the key points, the actual angle of the geometric figure formed by the key points). Because the imaging of the camera follows the principle of similar triangle, the geometric relationship on the imaging plane of the camera and the actual geometric relationship on the forklift tray entity are mutually corresponding, the actual depth from the forklift tray to the high-definition camera can be accurately calculated by utilizing the mutually corresponding geometric relationship and the camera focal length to perform similar calculation, and then calculation is performed according to the calculated actual depth and the positions of all key points in the forklift tray image according to the set proportion, a certain space point capable of representing the space position of the forklift tray is mapped in a camera coordinate system, and the 3D coordinate of the space point can be calculated, so that the position information of the forklift tray is determined, and the accurate positioning of the forklift tray is realized.
Compared with the technical scheme of a laser radar, the method reduces the use and maintenance cost, and compared with the technical scheme of a depth camera, the method avoids the defect that the distance is limited and is influenced by natural light. The scheme only needs to train a key point detection model, has small calculated amount, high running speed, simpler realization and easier popularization and application.
Specifically, in an embodiment, the forklift pallet jack is rectangular, and the step S102 specifically includes the following steps:
step one: and detecting a forklift pallet target from the forklift pallet image.
Step two: four vertexes corresponding to each forklift pallet jack are identified from the forklift pallet target through a preset key point detection model, and the identified vertexes are used as key points near the forklift pallet jacks.
Specifically, before key point detection is performed, the embodiment of the invention uses a high-resolution camera positioned on a forklift to collect a large amount of image data of different angles and different distances of a forklift pallet in a preparation stage. And labeling each image sample, as shown in fig. 5, labeling a 2D frame of the tray, a minimum circumscribed rectangle of the 2D frame, and key points of the tray (in the embodiment of the invention, four vertexes of jacks are labeled as key points for rectangular forklift pallet jacks, for example, the figure comprises two forklift pallet jacks, 8 key points in total, and one jack corresponds to the key points 1-4). Then, a tray target detection model1 and a key point detection model2 are constructed, and the two detection models are trained by using the marked image data. The target detection model1 can accurately identify a 2D frame of the forklift pallet (namely a forklift pallet target in a forklift pallet image), reduces the image range of the key point detection, and improves the accuracy of the key point detection. The key point detection model2 can accurately identify four vertexes of the jack from a 2D frame of the forklift pallet as key points.
For the target detection model1, an R-CNN series algorithm or a YOLO series algorithm may be adopted, and the specific implementation principle is not described herein in detail in the prior art. The model2 for detecting the key point can be divided into a top-down model and a bottom-up model, and the present embodiment is not limited to use of a specific key point detection method. Generally, a top-down method can obtain higher accuracy, including two steps of target detection and key point regression, where the regression process includes a method of directly regressing by using a regression network and a method of generating a thermodynamic diagram, specifically, for example, performing target detection on a key point by using a YOLO target detection algorithm, outputting a recognition result by using a key point regression method of a reset network followed by a full connection layer, or outputting a recognition result by using a method of heat map probability prediction based on a hret network. In practical application, for rectangular jacks, the number of key points is not limited to four vertexes, and the number of key points can be more, in the embodiment of the invention, the four vertexes of the jacks are at least used as the key points of each jack, so that the vertical geometric relationship and the transverse aggregate relationship of the jacks can be accurately extracted, and the position coordinates and the offset angles of the forklift tray in space can be respectively obtained in the subsequent calculation process based on the vertical geometric relationship and the transverse geometric relationship of the jacks, thereby further improving the accuracy of positioning the forklift tray.
Specifically, in an embodiment, the embodiment of the present invention optimizes the loss function of the training key point detection model, and proposes an improved loss function, where the improved loss function is:
L total ＝L i +L 2
wherein L is total Representing an improved overall loss function value, L 1 Represents the existing loss function value, L 2 Represents a loss function value calculated based on a key point near a pallet jack of a forklift, n represents the number of batches of training samples (for example, the training samples are divided into 20 batches of 100 samples), i represents the ith batch of training samples, K 1 [1]And K 2 [1]Respectively representing the x-axis coordinate of a first vertex and the x-axis coordinate of a second vertex positioned on the first vertical side of the current forklift pallet jack, K 3 [1]And K 4 [1]Respectively show the jacks positioned on the pallet of the current forkliftThe x-axis coordinates of the third vertex and the x-axis coordinates of the fourth vertex on the second vertical side.
Specifically, as shown in fig. 5, for a rectangular forklift pallet jack, the jack includes two vertical sides and two horizontal sides, and for one of the vertical sides, the two key points located on the vertical side in this embodiment are called a first vertex and a second vertex, for example, key points 1 and 2 in fig. 5. For another vertical edge, the present embodiment refers to two keypoints located on the vertical edge simultaneously as a third vertex and a fourth vertex, such as keypoints 3 and 4 in fig. 5. Theoretically, the x-coordinates of the first vertex and the second vertex should be the same, and the x-coordinates of the third vertex and the fourth vertex should be the same. Therefore, the present embodiment loses L at the present time 1 Based on common smooth average absolute error loss, mean square error loss and the like, adding a new loss term L with the aim of minimizing the difference between the x coordinates of the first vertex and the second vertex and the difference between the x coordinates of the third vertex and the fourth vertex 2 . The improved loss function of the embodiment of the invention trains the key point detection model, thereby further improving the accuracy of key point detection.
Specifically, in one embodiment, the step S103 specifically includes the following steps:
step three: and calculating the space coordinates corresponding to the pallet jacks of the forklift by utilizing the key points corresponding to the pallet jacks of the forklift and the camera internal parameters.
Step four: and determining position information according to the space coordinates corresponding to the pallet jacks of the forklift.
Specifically, according to the embodiment of the invention, for each forklift pallet jack, xy coordinates of key points nearby the forklift pallet jack are fused to obtain a fused xy coordinate, and the fused xy coordinate is used for representing the position of each forklift pallet jack in the xy plane position. Meanwhile, the focal length in the camera internal parameter is combined, the depth coordinates of the forklift tray jacks from the camera can be obtained by calculating based on the triangle-like principle, and the positions of each forklift tray jack in space can be accurately represented by combining the fused xy coordinates. Finally, the embodiment of the invention utilizes the space coordinates of each forklift pallet jack to represent the position information of the forklift pallet entity, so that the forklift can utilize the obtained position information to carry out path planning, the insertion of the pallet into the corresponding forklift pallet jack can be controlled more accurately, and the accuracy of automatic driving is further improved.
Specifically, in an embodiment, the third step specifically includes the following steps:
step five: and calculating an average value of xy coordinates of key points of the current forklift pallet jack in the image to obtain xy coordinates of a center point of the current forklift pallet jack.
Specifically, according to the embodiment of the invention, for each forklift pallet jack, the xy coordinates of key points nearby the forklift pallet jack are averaged, so that the xy coordinates of the center point which can represent the center position of the forklift pallet jack are obtained. Taking the rectangular forklift pallet jack in the above embodiment as an example, the key points are 4 vertexes of the jack, and the xy coordinates of the center point are calculated according to the following formula:
wherein K is C [:]Representing the center point coordinate of the jack of the forklift pallet, when K C [:]For K C [0]The y-coordinate of the center point is shown when K C [:]For K C [1]When representing the x-coordinate, K of the center point i [:]Then the coordinates of the ith key point are represented, when K i [:]For K i [0]The y-coordinate of the ith key point is represented when K i [:]For K i [1]And represents the x-coordinate of the ith keypoint.
For key points detected by the pallet jacks of the forklift in other shapes, the center point coordinates of the jacks can be calculated through the average value, and the principle is the same as that above, so that the description is omitted.
Step six: and measuring the physical distance between key points of the current forklift pallet jacks from the forklift pallet entity.
Specifically, the geometric parameters of each forklift pallet entity are measurable (such as the length, width and height of the forklift pallet), and in the embodiment of the invention, each key point is mapped onto the forklift pallet entity correspondingly in the training stage, and the entity distance (such as 50 cm, 60 cm, etc.) is measured on the forklift pallet entity for the connection line between the key points. The measured physical distance can be stored in the positioning system, and when the forklift truck recognizes key points in the image, the stored physical distance is directly read according to the converted proportion.
Step seven: based on the principle of similar triangles, the center point depth coordinate of the current forklift pallet jack is calculated by utilizing the xy coordinates of the camera internal parameters and the key points of the current forklift pallet jack in the image and the physical distance between the key points of the current forklift pallet jack.
Specifically, according to the embodiment of the invention, the image distance between key points in an image is obtained by calculating the xy coordinates of the key points in the image, then two key points are selected, similar triangle calculation can be performed by using the physical distance between the key points and the focal length parameters in the internal parameters of the camera, so that the actual depth of the current two key points from the camera in space is obtained, and the actual depth is converted in proportion to obtain the depth coordinate of the central point of the jack of the forklift tray. Taking the rectangular forklift pallet jack in the above embodiment as an example: selecting a key point 1 and a key point 2 in fig. 5, measuring the physical distance X between the key point 1 and the key point 2 on a forklift tray entity, calculating the image distance X' between the key point 1 and the key point 2 according to the xy coordinates of the key point 1 and the key point 2 on an image, and acquiring the focal length f of an internal camera reference, wherein the actual depth D of the key point 1 and the key point 2 from the camera in space is
Then, the actual depth D of the key point 1 and the key point 2 can be directly used as the actual depth D of the center point of the forklift tray jack c Or the actual depth D of the key point 1 and the actual depth D of the key point 2 are adjusted to obtain the actual depth D of the center point of the jack of the forklift pallet c The embodiment of the invention is not limited thereto. Finally, the actual depth D of the center point of the jack of the forklift pallet c Mapping to an image coordinate system to obtain the forklift palletAnd the z coordinate except the xy coordinate of the center point of the jack is used as the depth coordinate of the center point of the jack of the forklift pallet.
Step eight: and determining the space coordinates of the current forklift pallet jack by utilizing the depth coordinates of the central point of the current forklift pallet jack and the xy coordinates of the central point of the current forklift pallet jack.
Specifically, in this embodiment, the center point xyz coordinate of the forklift pallet jack is used as the space coordinate of the forklift pallet jack. Through the processing from the fifth step to the eighth step, the space coordinate of the center position of each forklift pallet jack is obtained through calculation, so that the fact that the forklift can accurately insert the fork into the center of each forklift pallet jack in the subsequent positioning stage is further guaranteed, and further the accuracy of forklift transfer forklift pallets and the reliability of automatic forklift driving are improved.
Specifically, in one embodiment, the step seven specifically includes the following steps:
step nine: and calculating the image distance between the key points of the current forklift pallet jacks by using xy coordinates of the key points of the current forklift pallet jacks in the image.
Step ten: and according to the corresponding relation between each image distance and the entity distance, carrying out similar triangle calculation by combining the focal length parameters in the internal parameters of the camera, and obtaining a plurality of depth information.
Step eleven: and calculating the average value of the depth information, and taking the obtained depth average value as the actual depth of the center point of the jack of the current forklift pallet.
Step twelve: and determining the depth coordinate of the center point of the jack of the current forklift pallet by utilizing the actual depth of the center point.
Specifically, in practical applications, the captured image of the forklift pallet may have a certain offset, and the forklift pallet in the image may not be completely parallel to the xy plane. In order to further improve accuracy of actual depth of a center point of a forklift pallet jack, the embodiment of the invention can calculate one actual depth by considering that every two key points, can calculate a plurality of actual depths for a plurality of key points near the forklift pallet jack in pairs, thereby obtaining a plurality of depth information, and finally calculates an average value for the plurality of depth informationThereby compensating the influence of the shot offset angle on the actual depth of the center point of the forklift pallet jack. For example: taking the rectangular forklift pallet jack in the above embodiment as an example, 4 vertices of each jack are detected as 4 key points, the embodiment of the present invention can connect the key point 1 and the key point 2 in fig. 5, calculate the depth information D1 of the key point 1 and the key point 2 according to the formula in the above step seven, and connect the key point 3 and the key point 4 in fig. 7, calculate the depth information D2 of the key point 3 and the key point 4 according to the formula in the above step seven, calculate the mean value of the depth information D1 and the depth information D2, and use the calculated mean value as the actual depth D of the center point c 。
Finally, the actual depth D of the center point of the jack of the forklift pallet c And mapping the depth coordinates to an image coordinate system to obtain more accurate depth coordinates of the center point of the forklift pallet jack.
The embodiment of the invention is only taken as an example, but not limited to, the number of the depth information participating in calculation is not limited to two, and the depth information can be flexibly adjusted along with the number of the key points, and the method for calculating the actual depth of the center point based on the average value provided by the embodiment is also applicable to forklift pallet jacks with other shapes.
Specifically, in an embodiment, the step four specifically includes the following steps:
step thirteen: and extracting two pieces of transverse depth information with different positions from the plurality of pieces of depth information in the transverse edge direction of the forklift pallet.
Step fourteen: and determining the transverse physical distance of the key point corresponding to the transverse depth information in the transverse edge direction of the forklift pallet.
Fifteen steps: and calculating the offset angle of the forklift pallet by using the inverse trigonometric function relation of the transverse depth information and the transverse entity distance.
Step sixteen: and taking the offset angle of the forklift pallet and the space coordinates corresponding to the jacks of each forklift pallet as position information.
Specifically, in the embodiment of the invention, the fact that a certain offset possibly exists in a shot forklift tray image is considered, the forklift tray in the image is not completely parallel to the xy plane, and the forklift tray is simply positioned by using the space coordinates corresponding to the forklift tray jacks, so that the inaccuracy in positioning still exists, and therefore, the offset angle of the forklift tray is calculated, and the accuracy of the positioning of the follow-up tray is further improved. Firstly, two pieces of transverse depth information with different positions are extracted from a plurality of pieces of depth information in the transverse direction of a forklift pallet, if the forklift pallet in fig. 5 is taken as an example, depth information D1 of a key point 1 and a key point 2 and depth information D2 of a key point 3 and a key point 4 can be selected as two pieces of transverse depth information, the directions of the depth information D1 to the depth information D2 are the transverse direction of the forklift pallet, when the forklift pallet deviates from an xy plane, the sizes of the depth information D1 and the depth information D2 are different, and the deviation angle can be calculated by utilizing the difference.
In this embodiment, the lateral physical distance between the key points corresponding to the two lateral depth information in the lateral direction of the pallet of the forklift, for example, the physical distance L between the key point 1 and the key point 3 is measured 1-3 And then calculating the offset angle of the forklift pallet by using the inverse trigonometric function relation between the transverse depth information and the measured transverse physical distance, wherein the formula is as follows:
in the formula, θ represents an offset angle of the forklift pallet.
The embodiment of the present invention is only exemplified, but not limited thereto. The calculation process can be executed as long as the connecting line direction of the key points corresponding to the two pieces of extracted depth information is the same as the transverse edge direction of the forklift pallet, so that the offset angle of the forklift pallet is obtained. And finally, taking the offset angle of the forklift pallet and the space coordinates corresponding to the jacks of each forklift pallet as position information, and effectively improving the accuracy of positioning the forklift pallet in the subsequent automatic driving stage.
Through the steps, the forklift tray to be positioned is shot by the forklift, and the forklift tray image is acquired; identifying key points near the forklift pallet jacks in the forklift pallet image; and reasoning the space position of the forklift pallet according to the identified key points and the camera internal parameters to obtain the position information used for representing the space position of the forklift pallet. Therefore, the scheme provided by the embodiment of the invention completes the 3D positioning of the tray position based on the camera equipment, reduces the use and maintenance cost compared with the laser radar technical scheme, and avoids the defects of limited distance and influence of natural light compared with the depth camera technical scheme. The scheme only needs to train a key point detection model, has small calculated amount, high running speed, simpler realization and easier popularization and application.
As shown in fig. 6, this embodiment further provides a device for extracting position information of a forklift pallet, where the device includes:
the image acquisition module 101 is used for shooting a forklift pallet to be positioned and acquiring forklift pallet images. For details, refer to the related description of step S101 in the above method embodiment, and no further description is given here.
The key point identification module 102 is configured to identify key points near the forklift pallet jack in the forklift pallet image. For details, refer to the related description of step S102 in the above method embodiment, and no further description is given here.
The tray positioning module 103 is configured to infer a spatial position of the forklift tray according to the identified key points and camera internal parameters, so as to obtain position information for characterizing the spatial position of the forklift tray, where the camera internal parameters are camera internal parameters for capturing images of the forklift tray. For details, see the description of step S103 in the above method embodiment, and the details are not repeated here.
The device for extracting the position information of the forklift pallet provided by the embodiment of the invention is used for executing the method for extracting the position information of the forklift pallet provided by the embodiment of the invention, the implementation mode is the same as the principle, and details are referred to the related description of the embodiment of the method and are not repeated.
Through the cooperation of the components, the forklift tray to be positioned is shot by the forklift, and a forklift tray image is obtained; identifying key points near the forklift pallet jacks in the forklift pallet image; and reasoning the space position of the forklift pallet according to the identified key points and the camera internal parameters to obtain the position information used for representing the space position of the forklift pallet. Therefore, the scheme provided by the embodiment of the invention completes the 3D positioning of the tray position based on the camera equipment, reduces the use and maintenance cost compared with the laser radar technical scheme, and avoids the defects of limited distance and influence of natural light compared with the depth camera technical scheme. The scheme only needs to train a key point detection model, has small calculated amount, high running speed, simpler realization and easier popularization and application.
Fig. 7 shows a domain controller according to an embodiment of the present invention, which at least includes a sensing processing unit 901, a decision processing unit 902, a control processing unit 903, and a communication unit 904, where the sensing processing unit 901, the decision processing unit 902, the control processing unit 903, and the communication unit 904 may be communicatively connected to each other by a bus or other manners, and in fig. 7, the bus manner is taken as an example.
In this embodiment, the sensing processing unit 901 and the decision processing unit 902 respectively include independent processors, and the sensing processing unit 901 and the decision processing unit 902 may respectively include independent memories, or may use shared memories.
In the embodiment of the present invention, the sensing processing unit 901 is mainly applied to a scene of an engineering machine, and mainly functions to perform sensing fusion processing on sensor data to obtain environmental information of an environment where the current engineering machine is located, and then send the environmental information to the control processing unit 903 or the decision processing unit 902 according to a data type of an environmental information signal.
The memory is used as a non-transitory computer readable storage medium for storing non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the methods in the method embodiments described above. The perception processing unit 901 executes various functional applications of the processor and data processing, i.e. implements the method in the above-described method embodiments, by running non-transitory software programs, instructions and modules stored in the memory.
The decision processing unit 902 functions as: after information fusion of surrounding environment, operation scene, vehicle state and the like is combined, a driving or operation strategy is formulated, and finally a control command is sent.
The main functions of the control processing unit 903 are: conversion between different types of signals of communication protocol conversion (CAN, ethernet, LIN, etc.), AD conversion (sensor input), DA conversion (control drive), etc. For example, the control processing unit 903 may be an MCU of a texas instrument TI chip TDA4VM, an MCU of an eye q series chip of israel mobileey company, an MCU of a japan rapeser-CAR chip R-CAR H3, an MCU of a chinese horizon company, or the like, which converts a signal scanned by the lidar into point cloud data.
The main functions of the communication unit 904 are: the wireless communication is performed, the communication mode includes but is not limited to 5G/4G network communication, wi-Fi communication and satellite communication, and the communication with the cloud server is performed, and the communication method mainly has the following effects: uploading the relevant state and information of the equipment to a cloud service, requesting a cloud server to assist in calculation processing, and downloading data through the cloud server to upgrade OTA software of the controller; communicating with nearby devices, the status of other devices can be received, and the job tasks can be completed cooperatively. The communication unit 110 of the control module may be a 5G module, a Wi-Fi module, a bluetooth module, etc.
It will be appreciated by those skilled in the art that implementing all or part of the above-described methods in the embodiments may be implemented by a computer program for instructing relevant hardware, and the implemented program may be stored in a computer readable storage medium, and the program may include the steps of the embodiments of the above-described methods when executed. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a Flash Memory (Flash Memory), a Hard Disk (HDD), or a Solid State Drive (SSD); the storage medium may also comprise a combination of memories of the kind described above.
Although embodiments of the present invention have been described in connection with the accompanying drawings, various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the invention, and such modifications and variations are within the scope of the invention as defined by the appended claims.
Claims (10)
1. The method for extracting the position information of the forklift pallet is characterized by comprising the following steps:
shooting a forklift pallet to be positioned, and acquiring a forklift pallet image;
identifying key points near forklift pallet jacks in the forklift pallet image;
and reasoning the space position of the forklift pallet according to the identified key points and the camera internal parameters to obtain the position information used for representing the space position of the forklift pallet, wherein the camera internal parameters are camera internal parameters for shooting the forklift pallet image.
2. The method of claim 1, wherein the forklift pallet jack is rectangular, the identifying key points in the forklift pallet image that are near the forklift pallet jack comprising:
detecting a forklift pallet target from the forklift pallet image;
and identifying four vertexes corresponding to each forklift pallet jack from the forklift pallet target through a preset key point detection model, and taking the identified vertexes as key points near the forklift pallet jacks.
3. The method of claim 2, wherein the keypoint detection model is trained by a modified loss function, wherein the modified loss function is:
L total ＝ 1 + 2
wherein L is total Representing an improved overall loss function value, L 1 Represents the existing loss function value, L 2 Represents a loss function value calculated based on key points near the forklift pallet jack, n represents the batch number of training samples, i represents the ith batch of training samples, K 1 [1]And K 2 [1]Respectively representing the x-axis coordinate of a first vertex and the x-axis coordinate of a second vertex positioned on the first vertical side of the current forklift pallet jack, L 3 [1]And K 4 [1]Respectively representing the x-axis coordinates of the third vertex and the fourth vertex located on the second vertical side of the current forklift pallet jack.
4. The method according to claim 1, wherein the reasoning about the spatial position of the forklift pallet according to the identified key points and the camera internal references, to obtain the position information for characterizing the spatial position of the forklift pallet, includes:
calculating space coordinates corresponding to each forklift pallet jack by utilizing key points corresponding to each forklift pallet jack and the camera internal parameters;
and determining the position information according to the space coordinates corresponding to the jack of each forklift pallet.
5. The method according to claim 4, wherein calculating the spatial coordinates corresponding to each forklift pallet jack using the keypoints corresponding to each forklift pallet jack and the camera parameters comprises:
calculating an average value of xy coordinates of key points of the current forklift pallet jack in the image to obtain xy coordinates of a center point of the current forklift pallet jack;
measuring the physical distance between key points of the current forklift pallet jacks from the forklift pallet entity;
based on a similar triangle principle, calculating the depth coordinate of the central point of the current forklift pallet jack by utilizing the xy coordinates of the camera internal parameters, the key points of the current forklift pallet jack in the image and the physical distance between the key points of the current forklift pallet jack;
and determining the space coordinates of the current forklift pallet jack by utilizing the depth coordinates of the central point of the current forklift pallet jack and the xy coordinates of the central point of the current forklift pallet jack.
6. The method of claim 5, wherein the calculating the center point depth coordinate of the current forklift pallet jack based on the similar triangle principle using the camera internal reference, the physical distance between the xy coordinates of the key point of the current forklift pallet jack in the image, and the key point of the current forklift pallet jack, comprises:
calculating the image distance between the key points of the current forklift pallet jacks by using xy coordinates of the key points of the current forklift pallet jacks in the image;
according to the corresponding relation between each image distance and the entity distance, carrying out similar triangle calculation by combining the focal length parameters in the camera internal parameters to obtain a plurality of depth information;
calculating the average value of the depth information, and taking the obtained average value of the depth as the actual depth of the center point of the current forklift pallet jack;
and determining the depth coordinate of the center point of the current forklift pallet jack by utilizing the actual depth of the center point.
7. The method of claim 6, wherein determining the location information based on the spatial coordinates corresponding to the respective forklift pallet jacks comprises:
extracting two pieces of lateral depth information with different positions from the depth information in the lateral side direction of the forklift pallet;
determining the transverse physical distance of the key point corresponding to the transverse depth information in the transverse edge direction of the forklift pallet;
calculating the offset angle of the forklift pallet by utilizing the inverse trigonometric function relation between the transverse depth information and the transverse entity distance;
and taking the offset angle of the forklift pallet and the space coordinates corresponding to the jacks of each forklift pallet as the position information.
8. A position information extraction device of a forklift pallet, the device comprising:
the image acquisition module is used for shooting a forklift pallet to be positioned and acquiring forklift pallet images;
the key point identification module is used for identifying key points near the forklift pallet jacks in the forklift pallet image;
the tray positioning module is used for reasoning the space position of the forklift tray according to the identified key points and the camera internal parameters, so as to obtain the position information used for representing the space position of the forklift tray, and the camera internal parameters are the camera internal parameters for shooting the forklift tray image.
9. A domain controller, comprising: a perception processing unit, a decision processing unit, a control processing unit and a communication unit, wherein the perception processing unit, the decision processing unit, the control processing unit and the communication unit are in communication connection with each other, computer instructions are stored in the perception processing unit, and the perception processing unit executes the computer instructions, thereby executing the method according to any one of claims 1-7.
10. A computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310165250.XA CN116468787A (en) | 2023-02-16 | 2023-02-16 | Position information extraction method and device of forklift pallet and domain controller |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310165250.XA CN116468787A (en) | 2023-02-16 | 2023-02-16 | Position information extraction method and device of forklift pallet and domain controller |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116468787A true CN116468787A (en) | 2023-07-21 |
Family
ID=87184772
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310165250.XA Pending CN116468787A (en) | 2023-02-16 | 2023-02-16 | Position information extraction method and device of forklift pallet and domain controller |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116468787A (en) |
-
2023
- 2023-02-16 CN CN202310165250.XA patent/CN116468787A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3407294B1 (en) | Information processing method, device, and terminal | |
| CN113657224B (en) | Method, device and equipment for determining object state in vehicle-road coordination | |
| CN111192331B (en) | External parameter calibration method and device for laser radar and camera | |
| CN109801333B (en) | Volume measurement method, device and system and computing equipment | |
| US20210342620A1 (en) | Geographic object detection apparatus and geographic object detection method | |
| CN111123242B (en) | Combined calibration method based on laser radar and camera and computer readable storage medium | |
| CN110969592B (en) | Image fusion method, automatic driving control method, device and equipment | |
| CN109784250A (en) | The localization method and device of automatically guiding trolley | |
| CN110956100A (en) | High-precision map generation method and device, electronic equipment and storage medium | |
| KR102490521B1 (en) | Automatic calibration through vector matching of the LiDAR coordinate system and the camera coordinate system | |
| CN112200911A (en) | Region overlapping type three-dimensional map construction method and device combined with markers | |
| Wang et al. | Autonomous landing of multi-rotors UAV with monocular gimbaled camera on moving vehicle | |
| CN113945937A (en) | Precision detection method, device and storage medium | |
| CN107504917B (en) | Three-dimensional size measuring method and device | |
| CN116452852A (en) | Automatic generation method of high-precision vector map | |
| CN112686951A (en) | Method, device, terminal and storage medium for determining robot position | |
| CN114545426A (en) | Positioning method, positioning device, mobile robot and computer readable medium | |
| CN112405526A (en) | Robot positioning method and device, equipment and storage medium | |
| CN116386373A (en) | Vehicle positioning method and device, storage medium and electronic equipment | |
| CN116736259A (en) | Laser point cloud coordinate calibration method and device for tower crane automatic driving | |
| CN116309882A (en) | Tray detection and positioning method and system for unmanned forklift application | |
| CN116309817A (en) | Tray detection and positioning method based on RGB-D camera | |
| CN116125489A (en) | Indoor object three-dimensional detection method, computer equipment and storage medium | |
| CN116468787A (en) | Position information extraction method and device of forklift pallet and domain controller | |
| CN111854678B (en) | Pose measurement method based on semantic segmentation and Kalman filtering under monocular vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |