WO2021016854A1 - Calibration method and device, movable platform, and storage medium - Google Patents

Calibration method and device, movable platform, and storage medium Download PDF

Info

Publication number
WO2021016854A1
WO2021016854A1 PCT/CN2019/098354 CN2019098354W WO2021016854A1 WO 2021016854 A1 WO2021016854 A1 WO 2021016854A1 CN 2019098354 W CN2019098354 W CN 2019098354W WO 2021016854 A1 WO2021016854 A1 WO 2021016854A1
Authority
WO
WIPO (PCT)
Prior art keywords
point cloud
cloud data
projected
dimensional space
data
Prior art date
Application number
PCT/CN2019/098354
Other languages
French (fr)
Chinese (zh)
Inventor
李威
刘天博
Original Assignee
深圳市大疆创新科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市大疆创新科技有限公司 filed Critical 深圳市大疆创新科技有限公司
Priority to CN201980030471.8A priority Critical patent/CN112106111A/en
Priority to PCT/CN2019/098354 priority patent/WO2021016854A1/en
Publication of WO2021016854A1 publication Critical patent/WO2021016854A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G06T7/85—Stereo camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/08—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Definitions

  • the present invention relates to the field of control technology, in particular to a calibration method, equipment, movable platform and storage medium.
  • the calibration methods between lidar and camera mainly include external parameter calibration with and without targets.
  • the target external parameter calibration method relies on specific markers such as calibration plates or labels, and the external parameter calibration process is mostly offline. This kind of method can achieve higher precision external parameter calibration under the condition of relying on specific markers, and the calibration results have good consistency.
  • the embodiment of the present invention provides a calibration method, equipment, a movable platform and a storage medium, which realizes the calibration of the surrounding environment of the movable platform when there is no specific marker, and improves the calibration accuracy.
  • an embodiment of the present invention provides a calibration method, which is applied to a movable platform on which a laser scanning device and a camera are provided, and the method includes:
  • an embodiment of the present invention provides a calibration device, including a memory and a processor
  • the memory is used to store programs
  • the processor is used to call the program, and when the program is executed, it is used to perform the following operations:
  • an embodiment of the present invention provides a movable platform, and the movable platform includes:
  • the power system configured on the fuselage is used to provide mobile power for the movable platform
  • an embodiment of the present invention provides a computer-readable storage medium that stores a computer program that, when executed by a processor, implements the method described in the first aspect.
  • the calibration device obtains the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera, and determines the second point cloud data according to the first point cloud data, and Project the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space.
  • the projected three-dimensional space Projecting onto the image data, and obtaining the optimal position of the projected three-dimensional space projected onto the image data, thereby achieving calibration of the surrounding environment of the movable platform when there is no specific marker, and improving the calibration accuracy.
  • Figure 1 is a schematic structural diagram of a calibration system provided by an embodiment of the present invention.
  • FIG. 2 is a schematic flowchart of a calibration method provided by an embodiment of the present invention.
  • FIG. 3 is a schematic diagram of a three-dimensional grid space provided by an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of a discontinuous point cloud provided by an embodiment of the present invention.
  • FIG. 5 is a schematic flowchart of an offline calibration method provided by an embodiment of the present invention.
  • FIG. 6 is a schematic flowchart of an online calibration method provided by an embodiment of the present invention.
  • Fig. 7 is a schematic structural diagram of a calibration device provided by an embodiment of the present invention.
  • the calibration method provided in the embodiment of the present invention may be executed by a calibration system, and specifically, may be executed by a calibration device in the calibration system.
  • the calibration system includes a calibration device and a movable platform.
  • the calibration device may be installed on a movable platform; in some embodiments, the calibration device may be spatially independent of the movable platform; in some embodiments, the calibration device The device may be a component of a movable platform, that is, the movable platform includes a calibration device.
  • the calibration method can also be applied to other mobile devices, such as mobile devices that can move autonomously, such as robots, unmanned vehicles, and unmanned ships.
  • the calibration equipment in the calibration system can obtain the first point cloud data corresponding to the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera; in some embodiments, the laser scanning device and the The cameras are respectively detachably connected to the movable platform. In other embodiments, the laser scanning device and the camera may also be fixedly arranged on the movable platform, which is not limited herein. Further, in some embodiments, the laser scanning device includes any one or more of laser radar, millimeter wave radar, and ultrasonic radar; in some embodiments, the first point cloud data may be Obtained through lidar acquisition, or acquired through millimeter wave radar, ultrasonic radar, etc. on a movable platform, which is not specifically limited in the embodiment of the present invention.
  • the lidar is a perceptual sensor that can obtain three-dimensional information of the scene.
  • the basic principle is to actively emit laser pulse signals to the detected object and obtain the reflected pulse signals.
  • the depth information of the distance detector of the object to be measured is calculated; Knowing the launch direction, obtain the angle information of the measured object relative to the lidar; combine the aforementioned depth and angle information to obtain a large number of detection points (called point clouds), and based on the point cloud, the spatial three-dimensional information of the measured object relative to the lidar can be reconstructed.
  • the invention provides a method for calibration of lidar and camera in a natural scene without relying on specific markers, and also provides a solution for online detection and correction of calibration results.
  • this solution can calibrate the camera and lidar offline; in some embodiments, the solution can also calibrate the camera and lidar online, and detect the calibration error between the lidar and the camera , To correct the calibration error to improve the calibration accuracy.
  • the calibration system provided by the embodiment of the present invention will be schematically described below with reference to FIG. 1.
  • FIG. 1 is a schematic structural diagram of a calibration system provided by an embodiment of the present invention.
  • the calibration system includes: a calibration device 11 and a movable platform 12.
  • a communication connection can be established between the movable platform 12 and the calibration device 11 through a wireless communication connection.
  • a communication connection between the movable platform 12 and the calibration device 11 may also be established through a wired communication connection.
  • the movable platform 12 may be a movable device such as an unmanned vehicle, an unmanned ship, and a movable robot.
  • the movable platform 12 includes a power system 121, and the power system 121 is used to provide the movable platform 12 with moving power.
  • the movable platform 12 and the calibration device 11 are independent of each other.
  • the calibration device 11 is set in a cloud server and establishes a communication connection with the movable platform 12 through a wireless communication connection.
  • the calibration device may obtain the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera, and determine the second point according to the first point cloud data Cloud data, where the second point cloud data is used to indicate invalid point cloud data and/or discontinuous point cloud data.
  • the calibration device can project the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space.
  • the The projected three-dimensional space is projected onto the image data collected by the camera, thereby obtaining the optimal position of the projected three-dimensional space projected on the image data, so as to realize a calibration method that does not rely on a calibration object and improve the calibration result Consistency.
  • FIG. 2 is a schematic flowchart of a calibration method provided by an embodiment of the present invention.
  • the method may be executed by a calibration device, and the specific explanation of the calibration device is as described above.
  • the method of the embodiment of the present invention includes the following steps.
  • S201 Acquire the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera.
  • the calibration equipment can obtain the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera.
  • the laser scanning device includes any one or more of laser radar, millimeter wave radar, and ultrasonic radar.
  • the camera may be mounted on a movable platform. In some embodiments, the camera can also be independent of the movable platform and installed in the environment where the movable platform is located. In some embodiments, the camera of the camera includes but is not limited to a binocular camera, a monocular camera, a TOF camera and other camera devices.
  • the calibration device may convert the first point cloud data into the camera coordinate system based on a preset conversion matrix to obtain the first point cloud data in the camera coordinate system corresponding to the surrounding environment where the movable platform is located.
  • Point cloud data wherein the preset conversion matrix includes an internal parameter matrix and an external parameter matrix, and the external parameter matrix includes a rotation matrix and/or a translation vector.
  • the external parameter matrix when the origin of the camera coordinate system is set on the movable platform, the external parameter matrix only includes a rotation matrix.
  • the internal parameter matrix is determined based on a plurality of internal parameters, and the internal parameters may be parameters of the camera, such as focal length, image principal point coordinates, and so on.
  • the external parameter matrix may be parameters calibrated by the camera and the laser scanning device. For example, it may include a rotation matrix and/or a translation vector, where the rotation matrix may be determined by the posture of the camera, The translation vector can be determined by the positioning information of the camera.
  • the calibration device may determine that the movable platform is in an offline low-speed state when the moving speed of the movable platform is less than a preset speed threshold, and obtain the data collected by the laser scanning device.
  • the first point cloud data of the surrounding environment and the image data collected by the camera when the mobile platform is in an offline low-speed state to achieve offline calibration. Through offline calibration, you can quickly collect enough calibration data at one time, reduce the impact of motion on the calibration accuracy, and improve the calibration accuracy.
  • the calibration device may establish a three-dimensional grid space relative to the camera coordinate system before acquiring the first point cloud data of the surrounding environment when the movable platform is offline and low-speed collected by the laser scanning device .
  • the first point cloud data may be projected to the image shown in FIG. 3 through external parameters.
  • Figure 3 is a schematic diagram of a three-dimensional grid space provided by an embodiment of the present invention.
  • the calibration equipment may determine that the movable platform is in a moving state when the moving speed of the movable platform is greater than or equal to the preset speed threshold, and obtain the data collected by the laser scanning device
  • the first point cloud data of the surrounding environment and the image data collected by the camera can be used to realize online error detection.
  • the calibration data that meets the requirements of a certain scene will be continuously collected, and the current calibration will be checked whether the current calibration is optimal. If a better calibration result is continuously found, the current calibration effect will be performed Update to ensure the consistency of calibration results.
  • S202 Determine second point cloud data according to the first point cloud data, where the second point cloud data is used to indicate invalid point cloud data and/or discontinuous point cloud data.
  • the calibration device may determine second point cloud data according to the first point cloud data, where the second point cloud data is used to indicate invalid point cloud data and/or discontinuous point cloud data.
  • the second point cloud data is used to indicate discontinuous point cloud data.
  • the calibration device may determine the distance between two adjacent first point cloud data in the first point cloud data, And according to the distance between the two adjacent first point cloud data, the discontinuous second point cloud data is determined.
  • the calibration device when the calibration device determines the discontinuous second point cloud data according to the distance between the two adjacent first point cloud data, it may determine that the two adjacent first point cloud data Whether the distance between the first point cloud data is greater than a first preset threshold, and when it is determined that the distance between the two adjacent first point cloud data is greater than the first preset threshold, determine the phase The two adjacent first point cloud data are discontinuous second point cloud data.
  • the data collected by the lidar is continuous. If the distance between the two point cloud data before and after changes greatly, it means that this is a place of depth jump, which is discontinuous.
  • Point cloud data For example, the distance between the two point clouds can be obtained through a suitable algorithm based on the depth information of the two point cloud data.
  • FIG. 4 is taken as an example for illustration.
  • FIG. 4 is a schematic diagram of a discontinuous point cloud provided by an embodiment of the present invention.
  • two adjacent first point cloud data point cloud 41 and point cloud 42 if it is determined that the distance between the point cloud 41 and the point cloud 42 is greater than the first preset threshold, it can be determined
  • the point cloud 41 and the point cloud 42 are discontinuous second point cloud data.
  • the first preset threshold may be a certain value.
  • the distance between the first point cloud data and the origin can also be obtained, and then the distance between the first point cloud data and the origin and the distance between two adjacent first point cloud data To determine whether the two adjacent point cloud data are discontinuous second point cloud data.
  • the first point cloud data whose distance from the origin is greater than the preset value can be determined, and from the first point cloud data whose distance from the origin is greater than the preset value, it is determined that two adjacent points Whether the distance between the first point cloud data is greater than a preset distance threshold.
  • the distance between two adjacent first point cloud data is greater than the preset distance threshold, it is determined that the two adjacent first point cloud data are discontinuous second point cloud data.
  • the distance between two adjacent first point cloud data whose distance to the origin is greater than the preset value may be set as the preset distance threshold.
  • the preset distance threshold may be a function related to the distance from the origin. For example, as the distance from the origin is farther, the preset distance threshold gradually increases, and as the distance from the origin is closer, the preset distance threshold slowing shrieking. In this way, the error caused by the divergence angle can be compensated, the probability of false detection can be reduced, and the calibration accuracy can be improved.
  • the second point cloud data is used to indicate invalid point cloud data.
  • the calibration device determines the second point cloud data according to the first point cloud data, it may determine whether depth information exists in the first point cloud data, and determine the first point cloud data according to the depth information.
  • the second point cloud data that is invalid in the point cloud data.
  • invalid point cloud data can be determined in a scene without radar echo.
  • the scene without radar echo includes sky, water, etc. in the background.
  • the calibration device when the calibration device determines the second point cloud data according to the depth information, it may determine from the first point cloud data that the first point cloud data without depth information is The invalid second point cloud data.
  • the lidar actively emits laser pulse signals to the detected object to obtain the reflected pulse signals.
  • the lidar collects the first point cloud data
  • the background of is the sky
  • the lidar cannot receive the pulse signal returned by the detected object, so the depth information of the first point cloud data cannot be obtained, so if the first point cloud data obtained If there is no depth information, it can be determined that the first point cloud data is invalid second point cloud data.
  • the calibration device when the calibration device determines the second point cloud data according to the depth information, it can acquire the change value of the depth information of the first point cloud data, when the first point cloud data When the change value of the depth information is greater than a second preset threshold, it is determined that the first point cloud data corresponding to the greater than the second preset threshold is invalid second point cloud data.
  • the background of the first point cloud data collected by the camera and lidar is scenes such as fences and grass. As the lidar passes through such fences and grasses, it will obtain a large amount of fluctuating depth information. It is invalid point cloud data.
  • the lidar passes through fences, grass, etc., a lot of first point cloud data is acquired. If the depth information of the acquired multiple first point cloud data is greater than the second preset threshold, at this time, the acquired If the depth information of the plurality of first point cloud data fluctuates greatly, it can be determined that the first point cloud data is invalid second point cloud data.
  • the second point cloud data is used to indicate invalid point cloud data and discontinuous point cloud data.
  • the calibration device is determining the second point cloud data according to the first point cloud data. The methods for invalid point cloud data and discontinuous point cloud data in the point cloud data are as described above, and will not be repeated here.
  • the calibration device may compare the acquired first point cloud data of the current frame with the acquired first point cloud data.
  • the cloud data is matched, and the degree of similarity between the spatial distribution of the first point cloud data of the current frame and the spatial distribution of the acquired first point cloud data is determined. If the similarity is greater than the preset similarity threshold, the calibration device may delete the first point cloud data of the current frame; if the similarity is less than or equal to the preset similarity threshold, it may determine to change The first point cloud data of the current frame is added to the first point cloud data that has been acquired.
  • the data of repeated scenes can be prevented from being repeatedly detected, so that the amount of invalid point cloud data can be reduced and the calculation efficiency can be improved.
  • the first point cloud data detected in each frame will be compared with the first point cloud data that has been acquired. If the spatial distribution is relatively similar, the first point cloud data of the frame will be deleted to ensure that each selected The first point cloud data of one frame can cover different scenes as much as possible.
  • S203 Project the second point cloud data to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space.
  • the calibration device may project the second point cloud data to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space.
  • the calibration device when the calibration device projects the second point cloud data to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space, it can determine the relative relationship between the laser scanning device and the camera. Position information, and project the second point cloud data to a three-dimensional grid space in the camera coordinate system according to the relative position information to obtain a projected three-dimensional space.
  • the calibration device may determine the second point before projecting the second point cloud data to the three-dimensional grid space in the camera coordinate system according to the relative position information to obtain the projected three-dimensional space
  • the cloud data is similar to the spatial distribution of the point cloud data that already exists in the three-dimensional grid space, and the second point cloud data whose spatial distribution similarity is greater than a preset similarity threshold is deleted. In this way, redundant point cloud data can be deleted in advance to improve computing efficiency.
  • the calibration device when the calibration device projects the second point cloud data to the three-dimensional raster space in the camera coordinate system according to the relative position information to obtain the projected three-dimensional space, it can be based on the relative position information Projecting the deleted second point cloud data to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space.
  • the calibration device may determine the Location information and location information of the point cloud data that already exists in the three-dimensional grid space, and based on the location information of the second point cloud data and the location information of the point cloud data that already exists in the three-dimensional grid space, Determine the spatial distribution similarity between the second point cloud data and the point cloud data that already exists in the three-dimensional grid space.
  • the calibration device before the calibration device projects the second point cloud data to the three-dimensional grid space in the camera coordinate system, it may be determined whether the angle of view of the camera is smaller than the angle of view of the laser scanning device. When the angle of view of the camera is smaller than the angle of view of the laser scanning device, the step of projecting the second point cloud data to the three-dimensional grid space in the camera coordinate system may be performed.
  • step S202 and step S203 can also be reversed.
  • the point cloud data can be projected into the three-dimensional raster space of the camera first, and then based on the first point cloud data
  • the second point cloud data is determined, and the second point cloud data is used to indicate invalid point cloud data and/or discontinuous point cloud data. This is only an exemplary description and is not limited herein.
  • the calibration device may project the projected three-dimensional space onto the image data collected by the camera, and obtain the projection The three-dimensional space is projected to the optimal position on the image data. Specifically, at the optimal position, the projection three-dimensional space matches the image data position optimally.
  • satisfying the preset condition includes that the quantity of the second point cloud data in each grid area in the projected three-dimensional space is greater than a preset quantity threshold.
  • the calibration device when the calibration device projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it can be based on The image data collected by the camera determines the gradient image corresponding to the image data, and projects the second point cloud data in the projected three-dimensional space onto the gradient image.
  • the calibration device can determine that the projected three-dimensional The optimal position of the spatial projection on the image data.
  • the calibration device when it is determined that the second point cloud data of the projected three-dimensional space is projected to the gradient image, and the second point cloud data of the projected three-dimensional space is completely fused with the gradient image, the calibration device
  • the optimal position of the projection three-dimensional space projected onto the image data can be determined according to the following formula (1).
  • D p is the gradient of the corresponding projection point on the image.
  • the calibration device when the calibration device determines the gradient image corresponding to the image data according to the image data collected by the camera, it may determine the gradient image corresponding to the image data according to the image data collected by the camera. And extracting gradient information and/or edge information from the gray image, so as to determine the gradient image according to the gradient information and/or edge information.
  • the calibration device when the calibration device projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it can obtain The projected three-dimensional space is projected onto the target image obtained from the image data collected by the camera, and the reflectivity of the second point cloud data in the target image is determined, and the gray scale of the grayscale image corresponding to the target image is determined. Degree value, so as to determine the optimal projected three-dimensional space projected onto the image data according to the reflectivity of the second point cloud data in the target image and the gray value of the gray image corresponding to the target image position.
  • the calibration device determines that the projection three-dimensional space is projected to the target image according to the reflectivity of the second point cloud data in the target image and the gray value of the gray image corresponding to the target image.
  • the optimal position of the projection three-dimensional space projected on the image data can be determined according to formula (2).
  • I p is the gray value of the corresponding projection point on the image.
  • the calibration device when the calibration device projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it can obtain The movement information of the movable platform in the moving process, and according to the movement information, the compensation information of the second point cloud data is determined, and the second point cloud data in the projected three-dimensional space is determined according to the compensation information Compensation is performed, so that the compensated second point cloud data is projected onto the image data collected by the camera to obtain an optimal position of the projection three-dimensional space projected onto the image data.
  • the motion information includes any one or more of position, speed information, and acceleration information.
  • the calibration device when the calibration device projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it can obtain The second point cloud data in the process of moving the movable platform within a preset time range, and the second point cloud data in the projection three-dimensional space acquired within the preset time range is projected to the camera On the collected image data to obtain the optimal position of the projection three-dimensional space projected onto the image data.
  • the image data collected by the camera is not limited to grayscale images, and this embodiment is only an exemplary description and is not limited herein.
  • the color image data collected by the camera can also be processed.
  • algorithms such as machine learning can be used to first identify specific objects in the scene, such as lane lines, telephone poles, etc., and determine that the projected three-dimensional space is projected to all objects based on physical information such as the reflectance and brightness of the identified specific objects. According to the optimal position on the image data, the probability of false detection can be reduced and the calibration accuracy can be improved.
  • the calibration device when the calibration device projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it can obtain During the movement of the movable platform, the projection three-dimensional space is projected onto multiple target images obtained by the image data collected by the camera, and the data of each target image is compared. If the data of each target image is determined If they are consistent, it can be determined that the position information of the target image is the optimal position projected onto the image data in the projected three-dimensional space.
  • the calibration device compares the data of each target image, if it is determined that the data of each target image is inconsistent, it can be determined that the external parameters of the laser scanning device have changed; further, The external parameters of the laser scanning device are updated.
  • the calibration device when it compares the data of each target image, if it is determined that the data of each target image is inconsistent, it can trigger a preset alarm device to give an alarm to remind the user of the laser
  • a preset alarm device to give an alarm to remind the user of the laser
  • the external parameters of the scanning device are changed, and further, the user may be prompted to check the laser scanning device or automatically check the laser scanning device, which is not limited here.
  • the calibration device takes the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera, and determines the second point cloud data according to the first point cloud data, and Project the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space.
  • the projected three-dimensional space Projecting onto the image data, and obtaining the optimal position of the projected three-dimensional space projected onto the image data, thereby achieving calibration of the surrounding environment of the movable platform when there is no specific marker, and improving the calibration accuracy.
  • FIG. 5 is a schematic flowchart of an offline calibration method provided by an embodiment of the present invention.
  • the first step of the surrounding environment of the movable platform is collected by lidar.
  • Point cloud data image data is collected by a camera, a point cloud depth discontinuity point is detected according to the first point cloud data, and the point cloud depth discontinuity point is determined to be the second point cloud data, and the second point cloud Project the data into the three-dimensional grid space to obtain the projected three-dimensional space, compare the projected three-dimensional space with the existing data, if it is similar, discard the frame data, if not, add the projected three-dimensional space data
  • a database when it is determined that the data in the database is sufficient, the optimal position of the projection three-dimensional space projected onto the image data is obtained.
  • Figure 6 is a schematic flow chart of an online calibration method provided by an embodiment of the present invention.
  • the online calibration process includes the offline calibration process.
  • the included offline calibration process will not be repeated here.
  • the difference between the online calibration process and the offline calibration process is that in the online calibration process, the projection three-dimensional space is obtained and projected onto the image data
  • the consistency check can be performed on the optimal position.
  • the consistency detection includes: storing the result of the optimal position in a result queue, and storing multiple optimal positions in the result queue to detect whether the optimal positions are consistent, And output the detection results, and judge whether the optimal position is consistent with the image data according to the detection results. If they are consistent, the external parameters have changed and the optimal position needs to be updated.
  • the structure may be loose and cannot Complete calibration. Further, when the optimal position is inconsistent with the image data, the preset alarm device can be triggered to give an alarm to remind the user that the external parameters of the laser scanning device have changed, or prompt the user to check the laser scanning device, or automatically
  • the laser scanning device performs inspection, which is not limited here.
  • FIG. 7 is a schematic structural diagram of a calibration device provided by an embodiment of the present invention.
  • the calibration device includes: a memory 701 and a processor 702.
  • the calibration device further includes a data interface 703, and the data interface 703 is used to transfer data information between the calibration device and other devices.
  • the memory 701 may include a volatile memory (volatile memory); the memory 701 may also include a non-volatile memory (non-volatile memory); the memory 701 may also include a combination of the foregoing types of memories.
  • the processor 702 may be a central processing unit (CPU).
  • the processor 702 may further include a hardware chip.
  • the aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof.
  • the foregoing PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), or any combination thereof.
  • the memory 701 is used to store a program, and the processor 702 can call the program stored in the memory 701 to perform the following steps:
  • processor 702 determines the second point cloud data according to the first point cloud data, it is specifically configured to:
  • the discontinuous second point cloud data is determined.
  • processor 702 determines the discontinuous second point cloud data according to the distance between the two adjacent first point cloud data, it is specifically configured to:
  • processor 702 is further configured to:
  • the discontinuous second point cloud data is determined according to the distance between the first point cloud data and the origin and the distance between the two adjacent first point cloud data.
  • the processor 702 determines the discontinuous second point according to the distance between the first point cloud data and the origin and the distance between the two adjacent first point cloud data When cloud data is used, it is specifically used for:
  • processor 702 determines the second point cloud data according to the first point cloud data, it is specifically configured to:
  • the invalid second point cloud data in the first point cloud data is determined according to the depth information.
  • processor 702 determines the second point cloud data according to the depth information, it is specifically configured to:
  • processor 702 determines the second point cloud data according to the depth information, it is specifically configured to:
  • the processor 702 is further configured to:
  • the similarity is less than or equal to the preset similarity threshold, it is determined to add the first point cloud data of the current frame to the first point cloud data that has been acquired.
  • processor 702 projects the second point cloud data to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space, it is specifically used for:
  • the second point cloud data is projected to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space.
  • the processor 702 projects the second point cloud data to the three-dimensional grid space in the camera coordinate system according to the relative position information, and before obtaining the projected three-dimensional space, it is also used to:
  • the processor 702 projects the second point cloud data to the three-dimensional grid space in the camera coordinate system according to the relative position information, when obtaining the projected three-dimensional space, it is specifically used for:
  • processor 702 determines the spatial distribution similarity between the second point cloud data and the point cloud data that already exists in the three-dimensional grid space, it is specifically configured to:
  • the position information of the second point cloud data and the position information of the point cloud data that already exists in the three-dimensional grid space determine the second point cloud data and the point cloud that already exists in the three-dimensional grid space The spatial distribution similarity of the data.
  • processor 702 projects the second point cloud data to the three-dimensional grid space in the camera coordinate system, it is also used to:
  • the step of projecting the second point cloud data to the three-dimensional grid space in the camera coordinate system is performed.
  • the meeting the preset condition includes:
  • the quantity of the second point cloud data in each grid area in the projected three-dimensional space is greater than a preset quantity threshold.
  • processor 702 projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it is specifically used for:
  • the processor 702 determines the gradient image corresponding to the image data according to the image data collected by the camera, it is specifically configured to:
  • the gradient image is determined according to the gradient information and/or edge information.
  • processor 702 projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it is specifically used for:
  • the optimal position of the projection three-dimensional space projected onto the image data is determined.
  • processor 702 projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it is specifically used for:
  • the motion information includes any one or more of position information, speed information, and acceleration information.
  • processor 702 projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it is specifically used for:
  • processor 702 projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it is specifically used for:
  • the position information of the target image is the optimal position of the projection three-dimensional space projected onto the image data.
  • processor 702 is further configured to:
  • processor 702 is further configured to:
  • a preset alarm device is triggered to give an alarm to prompt the user to check the laser scanning device.
  • the laser scanning device includes any one or more of laser radar, millimeter wave radar, and ultrasonic radar.
  • the calibration device obtains the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera, and determines the second point cloud data according to the first point cloud data, and Project the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space.
  • the projected three-dimensional space Projecting onto the image data, and obtaining the optimal position of the projected three-dimensional space projected onto the image data, thereby achieving calibration of the surrounding environment of the movable platform when there is no specific marker, and improving the calibration accuracy.
  • the embodiment of the present invention also provides a movable platform, the movable platform includes: a fuselage; a power system configured on the fuselage for providing mobile power for the movable platform; and the above-mentioned calibration equipment.
  • the movable platform obtains the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera, and determines the second point cloud data according to the first point cloud data, And projecting the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space.
  • the projected three-dimensional Space projection onto the image data When each grid area in the projected three-dimensional space meets a preset condition, the projected three-dimensional Space projection onto the image data, and obtain the optimal position of the projected three-dimensional space projected onto the image data, so as to realize the calibration of the surrounding environment of the movable platform when there is no specific marker, and improve the calibration accuracy .
  • the embodiment of the present invention also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method described in the embodiment corresponding to FIG. 2 of the present invention ,
  • the device corresponding to the embodiment of the present invention described in FIG. 7 can also be implemented, which will not be repeated here.
  • the computer-readable storage medium may be an internal storage unit of the device described in any of the foregoing embodiments, such as a hard disk or memory of the device.
  • the computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk equipped on the device, a Smart Media Card (SMC), or a Secure Digital (SD) card , Flash Card, etc.
  • the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device.
  • the computer-readable storage medium is used to store the computer program and other programs and data required by the terminal.
  • the computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

Abstract

The embodiments of the present invention provide a calibration method and device, a movable platform, and a storage medium. The method comprises: (S201) obtaining first point cloud data of surrounding environment of a movable platform collected by a laser scanning apparatus and image data collected by a camera; (S202) determining second point cloud data according to the first point cloud data, the second point cloud data being used for indicating invalid point cloud data and/or inconsecutive point cloud data; (S203) projecting the second point cloud data to a three-dimensional grid space under a camera coordinate system to obtain a projected three-dimensional space; and (S204) when each grid region in the projected three-dimensional space meets a preset condition, projecting the projected three-dimensional space onto the image data collected by the camera, and obtaining the optimal position of the projected three-dimensional space projected onto the image data. Therefore, the surrounding environment of the movable platform can be calibrated without a specific marker, and the calibration precision is improved.

Description

一种标定方法、设备、可移动平台及存储介质Calibration method, equipment, movable platform and storage medium 技术领域Technical field
本发明涉及控制技术领域，尤其涉及一种标定方法、设备、可移动平台及存储介质。The present invention relates to the field of control technology, in particular to a calibration method, equipment, movable platform and storage medium.
背景技术Background technique
目前，激光雷达和相机之间的标定方法主要包括有目标和无目标的外参标定。通常有目标的外参标定方法依赖于特定的标志物如标定板或者标签，且外参标定过程大多是离线的。这类方法可以在依赖特定的标志物的情况下实现较高精度的外参标定，且标定结果一致性较好。At present, the calibration methods between lidar and camera mainly include external parameter calibration with and without targets. Usually the target external parameter calibration method relies on specific markers such as calibration plates or labels, and the external parameter calibration process is mostly offline. This kind of method can achieve higher precision external parameter calibration under the condition of relying on specific markers, and the calibration results have good consistency.
然而，上述外参标定方法需要特定的标志物，标定过程较为繁琐，适用场景有限，不适合在室外进行标定，并且需要比较稠密的点云数据，因此对设备本身的性能要求较高。因此，如何实现在没有特定标志物时提高标定精度和标定结果的一致性成为研究的重点。However, the above external parameter calibration method requires specific markers, the calibration process is cumbersome, the applicable scenarios are limited, it is not suitable for outdoor calibration, and relatively dense point cloud data is required, so the performance requirements of the device itself are high. Therefore, how to improve the calibration accuracy and the consistency of the calibration results when there is no specific marker has become the focus of research.
发明内容Summary of the invention
本发明实施例提供了一种标定方法、设备、可移动平台及存储介质，实现了在没有特定标志物时对可移动平台周围环境的标定，提高了标定精度。The embodiment of the present invention provides a calibration method, equipment, a movable platform and a storage medium, which realizes the calibration of the surrounding environment of the movable platform when there is no specific marker, and improves the calibration accuracy.
第一方面，本发明实施例提供了一种标定方法，应用于可移动平台，所述可移动平台上设置了激光扫描装置和相机，所述方法包括：In the first aspect, an embodiment of the present invention provides a calibration method, which is applied to a movable platform on which a laser scanning device and a camera are provided, and the method includes:
获取激光扫描装置采集的所述可移动平台周围环境的第一点云数据以及相机采集的图像数据；Acquiring the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera;
根据所述第一点云数据确定第二点云数据，所述第二点云数据用于指示无效的点云数据和/或不连续的点云数据；Determining second point cloud data according to the first point cloud data, where the second point cloud data is used to indicate invalid point cloud data and/or discontinuous point cloud data;
将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间；Projecting the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space;
当所述投影三维空间中的每个栅格区域满足预设条件时，将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置。When each grid area in the projected three-dimensional space meets a preset condition, project the projected three-dimensional space onto the image data collected by the camera, and obtain the projected three-dimensional space and project it onto the image data The best location.
第二方面，本发明实施例提供了一种标定设备，包括存储器和处理器；In the second aspect, an embodiment of the present invention provides a calibration device, including a memory and a processor;
所述存储器，用于存储程序；The memory is used to store programs;
所述处理器，用于调用所述程序，当所述程序被执行时，用于执行以下操作：The processor is used to call the program, and when the program is executed, it is used to perform the following operations:
获取激光扫描装置采集的所述可移动平台周围环境的第一点云数据以及相机采集的图像数据；Acquiring the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera;
根据所述第一点云数据确定第二点云数据，所述第二点云数据用于指示无效的点云数据和/或不连续的点云数据；Determining second point cloud data according to the first point cloud data, where the second point cloud data is used to indicate invalid point cloud data and/or discontinuous point cloud data;
将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间；Projecting the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space;
当所述投影三维空间中的每个栅格区域满足预设条件时，将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置。When each grid area in the projected three-dimensional space meets a preset condition, project the projected three-dimensional space onto the image data collected by the camera, and obtain the projected three-dimensional space and project it onto the image data The best location.
第三方面，本发明实施例提供了一种可移动平台，所述可移动平台包括：In a third aspect, an embodiment of the present invention provides a movable platform, and the movable platform includes:
机身；body;
配置在机身上的动力系统，用于为所述可移动平台提供移动的动力；The power system configured on the fuselage is used to provide mobile power for the movable platform;
如上述第二方面所述的标定设备。The calibration device as described in the second aspect above.
第四方面，本发明实施例提供了一种计算机可读存储介质，该计算机可读存储介质存储有计算机程序，该计算机程序被处理器执行时实现如上述第一方面所述的方法。In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium that stores a computer program that, when executed by a processor, implements the method described in the first aspect.
本发明实施例中，标定设备通过获取激光扫描装置采集的可移动平台周围环境的第一点云数据以及相机采集的图像数据，并根据所述第一点云数据确定第二点云数据，以及将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间，当所述投影三维空间中的每个栅格区域满足预设条件时，将所述投影三维空间投影至所述图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置，从而实现了在没有特定标志物时对可移动平台周围环境进行标定，提高了标定精度。In the embodiment of the present invention, the calibration device obtains the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera, and determines the second point cloud data according to the first point cloud data, and Project the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space. When each grid area in the projected three-dimensional space meets a preset condition, the projected three-dimensional space Projecting onto the image data, and obtaining the optimal position of the projected three-dimensional space projected onto the image data, thereby achieving calibration of the surrounding environment of the movable platform when there is no specific marker, and improving the calibration accuracy.
附图说明Description of the drawings
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施 例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, without creative work, other drawings can be obtained from these drawings.
图1是本发明实施例提供的一种标定系统的结构示意图；Figure 1 is a schematic structural diagram of a calibration system provided by an embodiment of the present invention;
图2是本发明实施例提供的一种标定方法的流程示意图；2 is a schematic flowchart of a calibration method provided by an embodiment of the present invention;
图3是本发明实施例提供的一种三维栅格空间的示意图；3 is a schematic diagram of a three-dimensional grid space provided by an embodiment of the present invention;
图4是本发明实施例提供的一种不连续点云的示意图；4 is a schematic diagram of a discontinuous point cloud provided by an embodiment of the present invention;
图5是本发明实施例提供的一种离线标定方法的流程示意图；5 is a schematic flowchart of an offline calibration method provided by an embodiment of the present invention;
图6是本发明实施例提供的一种在线标定方法的流程示意图；6 is a schematic flowchart of an online calibration method provided by an embodiment of the present invention;
图7是本发明实施例提供的一种标定设备的结构示意图。Fig. 7 is a schematic structural diagram of a calibration device provided by an embodiment of the present invention.
具体实施方式Detailed ways
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。The technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.
下面结合附图，对本发明的一些实施方式作详细说明。在不冲突的情况下，下述的实施例及实施例中的特征可以相互组合。Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.
本发明实施例中提供的标定方法可以由一种标定系统执行，具体的，可以由标定系统中的标定设备执行。其中，所述标定系统包括标定设备和可移动平台。在某些实施例中，所述标定设备可以安装在可移动平台上；在某些实施例中，所述标定设备可以在空间上独立于可移动平台；在某些实施例中，所述标定设备可以是可移动平台的部件，即所述可移动平台包括标定设备。在其他实施例中，所述标定方法还可以应用于其他可移动设备上，如能够自主移动的机器人、无人车、无人船等可移动设备。The calibration method provided in the embodiment of the present invention may be executed by a calibration system, and specifically, may be executed by a calibration device in the calibration system. Wherein, the calibration system includes a calibration device and a movable platform. In some embodiments, the calibration device may be installed on a movable platform; in some embodiments, the calibration device may be spatially independent of the movable platform; in some embodiments, the calibration device The device may be a component of a movable platform, that is, the movable platform includes a calibration device. In other embodiments, the calibration method can also be applied to other mobile devices, such as mobile devices that can move autonomously, such as robots, unmanned vehicles, and unmanned ships.
所述标定系统中标定设备可以获取激光扫描装置采集的可移动平台所处周围环境对应的第一点云数据以及相机采集的图像数据；在某些实施例中，所述激光扫描装置和所述相机分别与所述可移动平台可拆卸连接，在其他实施例中，所述激光扫描装置和所述相机也可以固定设置在可移动平台，在此不作限定。进一步地，在某些实施例中，所述激光扫描装置包括激光雷达、毫米波雷 达、超声波雷达中的任意一种或多种；在某些实施例中，所述第一点云数据可以是通过激光雷达采集得到，也可以是通过可移动平台上的毫米波雷达、超声波雷达等获取得到，本发明实施例不做具体限定。The calibration equipment in the calibration system can obtain the first point cloud data corresponding to the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera; in some embodiments, the laser scanning device and the The cameras are respectively detachably connected to the movable platform. In other embodiments, the laser scanning device and the camera may also be fixedly arranged on the movable platform, which is not limited herein. Further, in some embodiments, the laser scanning device includes any one or more of laser radar, millimeter wave radar, and ultrasonic radar; in some embodiments, the first point cloud data may be Obtained through lidar acquisition, or acquired through millimeter wave radar, ultrasonic radar, etc. on a movable platform, which is not specifically limited in the embodiment of the present invention.
所述激光雷达是一种感知传感器，可以获得场景的三维信息。其基本原理为主动对被探测对象发射激光脉冲信号，并获得其反射回来的脉冲信号，根据发射信号和接收信号之间的时间差计算被测对象的距离探测器的深度信息；基于激光雷达的已知发射方向，获得被测对象相对激光雷达的角度信息；结合前述深度和角度信息得到海量的探测点(称为点云)，基于点云即可以重建被测对象相对激光雷达的空间三维信息。The lidar is a perceptual sensor that can obtain three-dimensional information of the scene. The basic principle is to actively emit laser pulse signals to the detected object and obtain the reflected pulse signals. According to the time difference between the transmitted signal and the received signal, the depth information of the distance detector of the object to be measured is calculated; Knowing the launch direction, obtain the angle information of the measured object relative to the lidar; combine the aforementioned depth and angle information to obtain a large number of detection points (called point clouds), and based on the point cloud, the spatial three-dimensional information of the measured object relative to the lidar can be reconstructed.
本发明提供了一种不依赖特定标志物在自然场景下的激光雷达和相机标定方法，也提供了在线检测标定结果并矫正的方案。通过点云在空间中分布的多样性，采集足够的数据，并利用多种点云特征和图像信息进行匹配得到标定精度更高的标定结果。在某些实施例中，本方案可以离线对相机和激光雷达进行标定；在某些实施例中，本方案也可以在线对相机和激光雷达进行标定，并检测激光雷达和相机之间的标定误差，以对该标定误差进行校正，提高标定精度。The invention provides a method for calibration of lidar and camera in a natural scene without relying on specific markers, and also provides a solution for online detection and correction of calibration results. Through the diversity of point cloud distribution in space, enough data is collected, and a variety of point cloud features and image information are used for matching to obtain calibration results with higher calibration accuracy. In some embodiments, this solution can calibrate the camera and lidar offline; in some embodiments, the solution can also calibrate the camera and lidar online, and detect the calibration error between the lidar and the camera , To correct the calibration error to improve the calibration accuracy.
下面结合附图1对本发明实施例提供的标定系统进行示意性说明。The calibration system provided by the embodiment of the present invention will be schematically described below with reference to FIG. 1.
请参见图1，图1是本发明实施例提供的一种标定系统的结构示意图。所述标定系统包括：标定设备11、可移动平台12。其中，可移动平台12和标定设备11之间可以通过无线通信连接方式建立通信连接。其中，在某些场景下，所述可移动平台12和标定设备11之间也可以通过有线通信连接方式建立通信连接。所述可移动平台12可以为无人车、无人船、可移动机器人等可移动设备。所述可移动平台12包括动力系统121，所述动力系统121用于为可移动平台12提供移动的动力。在其他实施例中，可移动平台12和标定设备11彼此独立，例如标定设备11设置在云端服务器中，通过无线通信连接方式与可移动平台12建立通信连接。Please refer to FIG. 1. FIG. 1 is a schematic structural diagram of a calibration system provided by an embodiment of the present invention. The calibration system includes: a calibration device 11 and a movable platform 12. Wherein, a communication connection can be established between the movable platform 12 and the calibration device 11 through a wireless communication connection. Among them, in some scenarios, a communication connection between the movable platform 12 and the calibration device 11 may also be established through a wired communication connection. The movable platform 12 may be a movable device such as an unmanned vehicle, an unmanned ship, and a movable robot. The movable platform 12 includes a power system 121, and the power system 121 is used to provide the movable platform 12 with moving power. In other embodiments, the movable platform 12 and the calibration device 11 are independent of each other. For example, the calibration device 11 is set in a cloud server and establishes a communication connection with the movable platform 12 through a wireless communication connection.
本发明实施例中，所述标定设备可以获取激光扫描装置采集的所述可移动平台周围环境的第一点云数据以及相机采集的图像数据，并根据所述第一点云数据确定第二点云数据，所述第二点云数据用于指示无效的点云数据和/或不 连续的点云数据。所述标定设备可以将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间，当所述投影三维空间中的每个栅格区域满足预设条件时，将所述投影三维空间投影至所述相机采集的图像数据上，从而获取所述投影三维空间投影到所述图像数据上的最优位置，以实现不依赖标定物的标定方法，并提高了标定结果的一致性。In the embodiment of the present invention, the calibration device may obtain the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera, and determine the second point according to the first point cloud data Cloud data, where the second point cloud data is used to indicate invalid point cloud data and/or discontinuous point cloud data. The calibration device can project the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space. When each grid area in the projected three-dimensional space meets a preset condition, the The projected three-dimensional space is projected onto the image data collected by the camera, thereby obtaining the optimal position of the projected three-dimensional space projected on the image data, so as to realize a calibration method that does not rely on a calibration object and improve the calibration result Consistency.
下面结合附图对本发明实施例提供的标定方法进行示意性说明。The calibration method provided by the embodiment of the present invention will be schematically described below with reference to the accompanying drawings.
具体请参见图2，图2是本发明实施例提供的一种标定方法的流程示意图，所述方法可以由标定设备执行，其中，标定设备的具体解释如前所述。具体地，本发明实施例的所述方法包括如下步骤。Please refer to FIG. 2 for details. FIG. 2 is a schematic flowchart of a calibration method provided by an embodiment of the present invention. The method may be executed by a calibration device, and the specific explanation of the calibration device is as described above. Specifically, the method of the embodiment of the present invention includes the following steps.
S201：获取激光扫描装置采集的可移动平台周围环境的第一点云数据以及相机采集的图像数据。S201: Acquire the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera.
本发明实施例中，标定设备可以获取激光扫描装置采集的所述可移动平台周围环境的第一点云数据以及相机采集的图像数据。In the embodiment of the present invention, the calibration equipment can obtain the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera.
在一些实施例中，所述激光扫描装置包括激光雷达、毫米波雷达、超声波雷达中的任意一种或多种。In some embodiments, the laser scanning device includes any one or more of laser radar, millimeter wave radar, and ultrasonic radar.
在一些实施例中，所述相机可以挂载在可移动平台上。在某些实施例中，所述相机还可以独立于可移动平台，安装于所述可移动平台所处环境当中。在某些实施例中，所述相机的摄像头包括但不限于双目摄像头、单目摄像头，TOF摄像头等摄像装置。In some embodiments, the camera may be mounted on a movable platform. In some embodiments, the camera can also be independent of the movable platform and installed in the environment where the movable platform is located. In some embodiments, the camera of the camera includes but is not limited to a binocular camera, a monocular camera, a TOF camera and other camera devices.
在一些实施例中，所述标定设备可以基于预设转换矩阵将所述第一点云数据转换到相机坐标系中，得到所述可移动平台所处周围环境对应的相机坐标系下的第一点云数据；其中，所述预设转换矩阵包括内参矩阵和外参矩阵，所述外参矩阵包括旋转矩阵和/或平移向量。在某些实施例中，当所述相机坐标系的原点设定在所述可移动平台上时，所述外参矩阵只包括旋转矩阵。In some embodiments, the calibration device may convert the first point cloud data into the camera coordinate system based on a preset conversion matrix to obtain the first point cloud data in the camera coordinate system corresponding to the surrounding environment where the movable platform is located. Point cloud data; wherein the preset conversion matrix includes an internal parameter matrix and an external parameter matrix, and the external parameter matrix includes a rotation matrix and/or a translation vector. In some embodiments, when the origin of the camera coordinate system is set on the movable platform, the external parameter matrix only includes a rotation matrix.
在某些实施例中，所述内参矩阵是根据多个内参数确定得到，所述内参数可以是相机的参数，如焦距、像主点坐标等。在某些实施例中，所述外参矩阵可以是相机和激光扫描装置标定得到的参数，例如可以包括旋转矩阵和/或平移向量，其中，所述旋转矩阵可以通过相机的姿态确定得到的，所述平移向量可以通过相机的定位信息确定得到。In some embodiments, the internal parameter matrix is determined based on a plurality of internal parameters, and the internal parameters may be parameters of the camera, such as focal length, image principal point coordinates, and so on. In some embodiments, the external parameter matrix may be parameters calibrated by the camera and the laser scanning device. For example, it may include a rotation matrix and/or a translation vector, where the rotation matrix may be determined by the posture of the camera, The translation vector can be determined by the positioning information of the camera.
在一个实施例中，所述标定设备可以在所述可移动平台的移动速度小于预设速度阈值时，此时可以确定所述可移动平台处于离线低速状态，并获取激光扫描装置采集的所述可移动平台处于离线低速状态时周围环境的第一点云数据以及相机采集的图像数据，以实现离线标定。通过离线标定可以快速一次性收集足够多的标定数据，减少运动对标定精度的影响，提高标定精度。In one embodiment, the calibration device may determine that the movable platform is in an offline low-speed state when the moving speed of the movable platform is less than a preset speed threshold, and obtain the data collected by the laser scanning device. The first point cloud data of the surrounding environment and the image data collected by the camera when the mobile platform is in an offline low-speed state to achieve offline calibration. Through offline calibration, you can quickly collect enough calibration data at one time, reduce the impact of motion on the calibration accuracy, and improve the calibration accuracy.
在一个实施例中，所述标定设备在获取激光扫描装置采集的所述可移动平台处于离线低速状态时周围环境的第一点云数据之前，可以建立一个相对于相机坐标系的三维栅格空间。所述标定设备在获取激光扫描装置采集的所述可移动平台处于离线低速状态时周围环境的第一点云数据之后，可以通过外参将所述第一点云数据投影到如图3所示的相机坐标系下的三维栅格空间，图3是本发明实施例提供的一种三维栅格空间的示意图。当所述三维点云空间中的第一点云数据的数量大于预设数量阈值时，确定离线低速采集到了足够多的点云数据，并执行步骤S202。In one embodiment, the calibration device may establish a three-dimensional grid space relative to the camera coordinate system before acquiring the first point cloud data of the surrounding environment when the movable platform is offline and low-speed collected by the laser scanning device . After the calibration device acquires the first point cloud data of the surrounding environment when the movable platform is in an offline low-speed state collected by the laser scanning device, the first point cloud data may be projected to the image shown in FIG. 3 through external parameters. Figure 3 is a schematic diagram of a three-dimensional grid space provided by an embodiment of the present invention. When the number of first point cloud data in the three-dimensional point cloud space is greater than the preset number threshold, it is determined that sufficient point cloud data has been collected offline at low speed, and step S202 is executed.
在一个实施例中，所述标定设备可以在所述可移动平台的移动速度大于或等于所述预设速度阈值时，确定所述可移动平台处于运动状态，并获取激光扫描装置采集的所述可移动平台处于运动状态时周围环境的第一点云数据以及相机采集的图像数据，以实现在线误差检测。通过在线误差检测会在可移动平台的运动过程中，持续采集符合一定场景要求的标定数据，并检测当前标定是否最优，如果持续发现有更优的标定结果，则会对当前的标定效果进行更新，从而确保标定结果的一致性。In one embodiment, the calibration equipment may determine that the movable platform is in a moving state when the moving speed of the movable platform is greater than or equal to the preset speed threshold, and obtain the data collected by the laser scanning device When the movable platform is in motion, the first point cloud data of the surrounding environment and the image data collected by the camera can be used to realize online error detection. Through online error detection, during the movement of the movable platform, the calibration data that meets the requirements of a certain scene will be continuously collected, and the current calibration will be checked whether the current calibration is optimal. If a better calibration result is continuously found, the current calibration effect will be performed Update to ensure the consistency of calibration results.
S202：根据所述第一点云数据确定第二点云数据，所述第二点云数据用于指示无效的点云数据和/或不连续的点云数据。S202: Determine second point cloud data according to the first point cloud data, where the second point cloud data is used to indicate invalid point cloud data and/or discontinuous point cloud data.
本发明实施例中，标定设备可以根据所述第一点云数据确定第二点云数据，所述第二点云数据用于指示无效的点云数据和/或不连续的点云数据。In the embodiment of the present invention, the calibration device may determine second point cloud data according to the first point cloud data, where the second point cloud data is used to indicate invalid point cloud data and/or discontinuous point cloud data.
在一个实施例中，所述第二点云数据用于指示不连续的点云数据。具体地，所述标定设备在根据所述第一点云数据确定第二点云数据时，可以确定所述第一点云数据中相邻两个所述第一点云数据之间的距离，并根据所述相邻两个所述第一点云数据之间的距离，确定不连续的第二点云数据。In one embodiment, the second point cloud data is used to indicate discontinuous point cloud data. Specifically, when determining the second point cloud data according to the first point cloud data, the calibration device may determine the distance between two adjacent first point cloud data in the first point cloud data, And according to the distance between the two adjacent first point cloud data, the discontinuous second point cloud data is determined.
在一个实施例中，所述标定设备在根据所述相邻两个所述第一点云数据之间的距离，确定不连续的第二点云数据时，可以确定所述相邻两个所述第一点 云数据之间的距离是否大于第一预设阈值，当确定出所述相邻两个所述第一点云数据之间的距离大于第一预设阈值时，确定所述相邻两个所述第一点云数据为不连续的第二点云数据。In one embodiment, when the calibration device determines the discontinuous second point cloud data according to the distance between the two adjacent first point cloud data, it may determine that the two adjacent first point cloud data Whether the distance between the first point cloud data is greater than a first preset threshold, and when it is determined that the distance between the two adjacent first point cloud data is greater than the first preset threshold, determine the phase The two adjacent first point cloud data are discontinuous second point cloud data.
具体实施例中，由于激光雷达的工作特性，激光雷达采集的数据是连续的，如果前后两个点云数据的距离发生了较大的变化，说明这是一个深度跳变的地方，属于不连续的点云数据。例如，可以根据两个点云数据的深度信息，通过合适的算法得到两个点云之间的距离。In the specific embodiment, due to the working characteristics of the lidar, the data collected by the lidar is continuous. If the distance between the two point cloud data before and after changes greatly, it means that this is a place of depth jump, which is discontinuous. Point cloud data. For example, the distance between the two point clouds can be obtained through a suitable algorithm based on the depth information of the two point cloud data.
具体可以图4为例进行举例说明，图4是本发明实施例提供的一种不连续点云的示意图。如图4所述，相邻两个第一点云数据点云41和点云42，如果确定出所述点云41和点云42之间的距离大于第一预设阈值时，则可以确定所述点云41和点云42为不连续的第二点云数据。例如，该第一预设阈值可以为一定值。Specifically, FIG. 4 is taken as an example for illustration. FIG. 4 is a schematic diagram of a discontinuous point cloud provided by an embodiment of the present invention. As shown in Figure 4, two adjacent first point cloud data point cloud 41 and point cloud 42, if it is determined that the distance between the point cloud 41 and the point cloud 42 is greater than the first preset threshold, it can be determined The point cloud 41 and the point cloud 42 are discontinuous second point cloud data. For example, the first preset threshold may be a certain value.
在另一个实施例中，还可以获取第一点云数据和原点之间的距离，进而通过第一点云数据和原点之间的距离、以及相邻两个第一点云数据之间的距离，确定该相邻的两个点云数据是否为不连续的第二点云数据。具体地，可以确定与原点之间的距离大于预设值的第一点云数据，并从所述与原点之间的距离大于预设值的第一点云数据中，确定相邻两个所述第一点云数据之间的距离是否大于预设距离阈值。当相邻两个第一点云数据之间的距离大于预设距离阈值时，确定该相邻两个第一点云数据为不连续的第二点云数据。在某些实施例中，由于受到发散角的影响，可以将距离原点大于预设值的相邻两个第一点云数据之间的距离设为预设距离阈值。在一种实施方式中，该预设距离阈值可以为与原点的距离相关的函数，例如，当距离原点越远，该预设距离阈值逐渐增大，当距离原点越近，该预设距离阈值逐渐减小。如此，可以补偿发散角所引起的误差，减小误检的概率，提高标定精度。In another embodiment, the distance between the first point cloud data and the origin can also be obtained, and then the distance between the first point cloud data and the origin and the distance between two adjacent first point cloud data To determine whether the two adjacent point cloud data are discontinuous second point cloud data. Specifically, the first point cloud data whose distance from the origin is greater than the preset value can be determined, and from the first point cloud data whose distance from the origin is greater than the preset value, it is determined that two adjacent points Whether the distance between the first point cloud data is greater than a preset distance threshold. When the distance between two adjacent first point cloud data is greater than the preset distance threshold, it is determined that the two adjacent first point cloud data are discontinuous second point cloud data. In some embodiments, due to the influence of the divergence angle, the distance between two adjacent first point cloud data whose distance to the origin is greater than the preset value may be set as the preset distance threshold. In an embodiment, the preset distance threshold may be a function related to the distance from the origin. For example, as the distance from the origin is farther, the preset distance threshold gradually increases, and as the distance from the origin is closer, the preset distance threshold slowing shrieking. In this way, the error caused by the divergence angle can be compensated, the probability of false detection can be reduced, and the calibration accuracy can be improved.
在一个实施例中，所述第二点云数据用于指示无效的点云数据。具体地，所述标定设备在根据所述第一点云数据确定第二点云数据时，可以确定所述第一点云数据中是否存在深度信息，并根据所述深度信息确定所述第一点云数据中为无效的所述第二点云数据。通过这种实施方式可以在没有雷达回波的场景下确定出无效的点云数据。在某些实施例中，所述没有雷达回波的场景包括背景为天空、水域等。In one embodiment, the second point cloud data is used to indicate invalid point cloud data. Specifically, when the calibration device determines the second point cloud data according to the first point cloud data, it may determine whether depth information exists in the first point cloud data, and determine the first point cloud data according to the depth information. The second point cloud data that is invalid in the point cloud data. Through this embodiment, invalid point cloud data can be determined in a scene without radar echo. In some embodiments, the scene without radar echo includes sky, water, etc. in the background.
在一个实施例中，所述标定设备在根据所述深度信息确定所述第二点云数据时，可以从所述第一点云数据中确定不存在深度信息的所述第一点云数据为无效的所述第二点云数据。In an embodiment, when the calibration device determines the second point cloud data according to the depth information, it may determine from the first point cloud data that the first point cloud data without depth information is The invalid second point cloud data.
例如，假设相机和激光雷达采集第一点云数据的背景为天空，由于激光雷达是主动对被探测对象发射激光脉冲信号，来获得其反射回来的脉冲信号，当激光雷达采集第一点云数据的背景为天空时，天空中没有被探测对象，因此激光雷达接收不到被探测对象返回的脉冲信号，从而获取不到第一点云数据的深度信息，因此如果获取到的第一点云数据不存在深度信息，则可以确定该第一点云数据为无效的第二点云数据。For example, suppose that the background of the first point cloud data collected by the camera and the lidar is the sky, because the lidar actively emits laser pulse signals to the detected object to obtain the reflected pulse signals. When the lidar collects the first point cloud data When the background of is the sky, there is no detected object in the sky, so the lidar cannot receive the pulse signal returned by the detected object, so the depth information of the first point cloud data cannot be obtained, so if the first point cloud data obtained If there is no depth information, it can be determined that the first point cloud data is invalid second point cloud data.
在一个实施例中，所述标定设备在根据所述深度信息确定所述第二点云数据时，可以获取所述第一点云数据的深度信息的变化值，当所述第一点云数据的深度信息的变化值大于第二预设阈值时，确定所述大于第二预设阈值对应的所述第一点云数据为无效的所述第二点云数据。In one embodiment, when the calibration device determines the second point cloud data according to the depth information, it can acquire the change value of the depth information of the first point cloud data, when the first point cloud data When the change value of the depth information is greater than a second preset threshold, it is determined that the first point cloud data corresponding to the greater than the second preset threshold is invalid second point cloud data.
例如，假设相机和激光雷达采集第一点云数据的背景为诸如栅栏、草丛等场景时，由于激光雷达穿过诸如栅栏、草丛等时会获取到大量波动较大的深度信息，此类深度信息为无效的点云数据。当激光雷达穿过诸如栅栏、草丛等时获取到许多第一点云数据，如果获取到的多个第一点云数据的深度信息的变化值均大于第二预设阈值，此时，获取到的多个第一点云数据的深度信息波动较大，则可以确定该第一点云数据为无效的第二点云数据。For example, suppose that the background of the first point cloud data collected by the camera and lidar is scenes such as fences and grass. As the lidar passes through such fences and grasses, it will obtain a large amount of fluctuating depth information. It is invalid point cloud data. When the lidar passes through fences, grass, etc., a lot of first point cloud data is acquired. If the depth information of the acquired multiple first point cloud data is greater than the second preset threshold, at this time, the acquired If the depth information of the plurality of first point cloud data fluctuates greatly, it can be determined that the first point cloud data is invalid second point cloud data.
在另一种实施例中，所述第二点云数据用于指示无效的点云数据和不连续的点云数据，具体地，所述标定设备在根据所述第一点云数据确定第二点云数据中的无效的点云数据和不连续的点云数据的方法如上所述，在此不再赘述。In another embodiment, the second point cloud data is used to indicate invalid point cloud data and discontinuous point cloud data. Specifically, the calibration device is determining the second point cloud data according to the first point cloud data. The methods for invalid point cloud data and discontinuous point cloud data in the point cloud data are as described above, and will not be repeated here.
在一个实施例中，所述标定设备在根据所述第一点云数据确定第二点云数据之前，可以将获取到的当前帧的第一点云数据与已经获取到的所述第一点云数据进行匹配，并确定所述当前帧的第一点云数据的空间分布与所述已经获取到的所述第一点云数据的空间分布的相似度。如果所述相似度大于预设相似度阈值，则所述标定设备可以删除所述当前帧的第一点云数据；如果所述相似度小于或等于所述预设相似度阈值，则可以确定将所述当前帧的第一点云数据加入已经获取到的所述第一点云数据。In one embodiment, before determining the second point cloud data according to the first point cloud data, the calibration device may compare the acquired first point cloud data of the current frame with the acquired first point cloud data. The cloud data is matched, and the degree of similarity between the spatial distribution of the first point cloud data of the current frame and the spatial distribution of the acquired first point cloud data is determined. If the similarity is greater than the preset similarity threshold, the calibration device may delete the first point cloud data of the current frame; if the similarity is less than or equal to the preset similarity threshold, it may determine to change The first point cloud data of the current frame is added to the first point cloud data that has been acquired.
可见，通过这种实施方式，可以避免重复场景的数据被反复检测出来，从 而可以减小无效点云数据的数据量，提高计算效率。对于每一帧检测出来的第一点云数据会和已经获取到的第一点云数据进行比较，如果空间分布比较相似，则将该帧的第一点云数据删除，从而确保挑选出来的每一帧的第一点云数据能够尽可能覆盖不同的场景。It can be seen that through this embodiment, the data of repeated scenes can be prevented from being repeatedly detected, so that the amount of invalid point cloud data can be reduced and the calculation efficiency can be improved. The first point cloud data detected in each frame will be compared with the first point cloud data that has been acquired. If the spatial distribution is relatively similar, the first point cloud data of the frame will be deleted to ensure that each selected The first point cloud data of one frame can cover different scenes as much as possible.
S203：将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间。S203: Project the second point cloud data to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space.
本发明实施例中，标定设备可以将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间。In the embodiment of the present invention, the calibration device may project the second point cloud data to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space.
在一个实施例中，所述标定设备在将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间时，可以确定所述激光扫描装置和相机之间的相对位置信息，并根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间。In an embodiment, when the calibration device projects the second point cloud data to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space, it can determine the relative relationship between the laser scanning device and the camera. Position information, and project the second point cloud data to a three-dimensional grid space in the camera coordinate system according to the relative position information to obtain a projected three-dimensional space.
在一个实施例中，所述标定设备在根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间之前，可以确定所述第二点云数据与所述三维栅格空间中已存在的点云数据的空间分布相似度，并删除所述空间分布相似度大于预设相似度阈值的第二点云数据。如此，可以预先删除冗余的点云数据，提高运算效率。In one embodiment, the calibration device may determine the second point before projecting the second point cloud data to the three-dimensional grid space in the camera coordinate system according to the relative position information to obtain the projected three-dimensional space The cloud data is similar to the spatial distribution of the point cloud data that already exists in the three-dimensional grid space, and the second point cloud data whose spatial distribution similarity is greater than a preset similarity threshold is deleted. In this way, redundant point cloud data can be deleted in advance to improve computing efficiency.
在一个实施例中，所述标定设备在根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间时，可以根据所述相对位置信息将所述删除后的第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间。In one embodiment, when the calibration device projects the second point cloud data to the three-dimensional raster space in the camera coordinate system according to the relative position information to obtain the projected three-dimensional space, it can be based on the relative position information Projecting the deleted second point cloud data to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space.
在一个实施例中，所述标定设备在确定所述第二点云数据与所述三维栅格空间中已存在的点云数据的空间分布相似度时，可以确定所述第二点云数据的位置信息以及所述三维栅格空间中已存在的点云数据的位置信息，并根据所述第二点云数据的位置信息与所述三维栅格空间中已存在的点云数据的位置信息，确定所述第二点云数据与所述三维栅格空间中已存在的点云数据的空间分布相似度。In one embodiment, when the calibration device determines the spatial distribution similarity between the second point cloud data and the point cloud data that already exists in the three-dimensional grid space, the calibration device may determine the Location information and location information of the point cloud data that already exists in the three-dimensional grid space, and based on the location information of the second point cloud data and the location information of the point cloud data that already exists in the three-dimensional grid space, Determine the spatial distribution similarity between the second point cloud data and the point cloud data that already exists in the three-dimensional grid space.
在一个实施例中，所述标定设备将所述第二点云数据投影至相机坐标系下的三维栅格空间之前，可以确定所述相机的视角是否小于所述激光扫描装置的视角，当确定所述相机的视角小于所述激光扫描装置的视角时，可以执行所述 将所述第二点云数据投影至相机坐标系下的三维栅格空间的步骤。In one embodiment, before the calibration device projects the second point cloud data to the three-dimensional grid space in the camera coordinate system, it may be determined whether the angle of view of the camera is smaller than the angle of view of the laser scanning device. When the angle of view of the camera is smaller than the angle of view of the laser scanning device, the step of projecting the second point cloud data to the three-dimensional grid space in the camera coordinate system may be performed.
可以理解，在一种实施例中，也可以将步骤S202和步骤S203的先后顺序进行调换，例如，可以先把点云数据投影到相机的三维栅格空间，然后根据所述第一点云数据确定第二点云数据，所述第二点云数据用于指示无效的点云数据和/或不连续的点云数据，此处仅为示例性说明，在此不作限定。It can be understood that, in an embodiment, the sequence of step S202 and step S203 can also be reversed. For example, the point cloud data can be projected into the three-dimensional raster space of the camera first, and then based on the first point cloud data The second point cloud data is determined, and the second point cloud data is used to indicate invalid point cloud data and/or discontinuous point cloud data. This is only an exemplary description and is not limited herein.
S204：当所述投影三维空间中的每个栅格区域满足预设条件时，将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置。S204: When each grid area in the projected three-dimensional space satisfies a preset condition, project the projected three-dimensional space onto the image data collected by the camera, and obtain the projected three-dimensional space projected onto the image The best position on the data.
本发明实施例中，当所述投影三维空间中的每个栅格区域满足预设条件时，标定设备可以将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置。具体地，在该最优位置上，该投影三维空间与该图像数据位置匹配最优。In the embodiment of the present invention, when each grid area in the projected three-dimensional space satisfies a preset condition, the calibration device may project the projected three-dimensional space onto the image data collected by the camera, and obtain the projection The three-dimensional space is projected to the optimal position on the image data. Specifically, at the optimal position, the projection three-dimensional space matches the image data position optimally.
在某些实施例中，所述满足预设条件，包括所述投影三维空间中的每个栅格区域中的第二点云数据的数量大于预设数量阈值。In some embodiments, satisfying the preset condition includes that the quantity of the second point cloud data in each grid area in the projected three-dimensional space is greater than a preset quantity threshold.
在一个实施例中，所述标定设备在将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，可以根据所述相机采集的图像数据，确定与所述图像数据对应的梯度图像，并将所述投影三维空间的第二点云数据投影至所述梯度图像。当确定所述投影三维空间的第二点云数据投影至所述梯度图像，所述投影三维空间的第二点云数据与所述梯度图像完全融合时，所述标定设备可以确定所述投影三维空间投影到所述图像数据上的最优位置。In one embodiment, when the calibration device projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it can be based on The image data collected by the camera determines the gradient image corresponding to the image data, and projects the second point cloud data in the projected three-dimensional space onto the gradient image. When it is determined that the second point cloud data of the projected three-dimensional space is projected to the gradient image, and the second point cloud data of the projected three-dimensional space is completely fused with the gradient image, the calibration device can determine that the projected three-dimensional The optimal position of the spatial projection on the image data.
在一个实施例中，当确定所述投影三维空间的第二点云数据投影至所述梯度图像，所述投影三维空间的第二点云数据与所述梯度图像完全融合时，所述标定设备可以根据如下公式(1)确定所述投影三维空间投影到所述图像数据上的最优位置。In one embodiment, when it is determined that the second point cloud data of the projected three-dimensional space is projected to the gradient image, and the second point cloud data of the projected three-dimensional space is completely fused with the gradient image, the calibration device The optimal position of the projection three-dimensional space projected onto the image data can be determined according to the following formula (1).
Figure PCTCN2019098354-appb-000001
Figure PCTCN2019098354-appb-000001
其中，D p为图像上对应投影点的梯度。 Among them, D p is the gradient of the corresponding projection point on the image.
在一个实施例中，所述标定设备在根据所述相机采集的图像数据，确定与所述图像数据对应的梯度图像时，可以根据所述相机采集的图像数据，确定与 所述图像数据对应的灰度图像，并从所述灰度图像中提取梯度信息和/或边缘信息，从而根据所述梯度信息和/或边缘信息，确定所述梯度图像。In one embodiment, when the calibration device determines the gradient image corresponding to the image data according to the image data collected by the camera, it may determine the gradient image corresponding to the image data according to the image data collected by the camera. And extracting gradient information and/or edge information from the gray image, so as to determine the gradient image according to the gradient information and/or edge information.
在一个实施例中，所述标定设备将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，可以获取将所述投影三维空间投影至所述相机采集的图像数据上得到的目标图像，并确定所述目标图像中第二点云数据的反射率，以及确定与所述目标图像对应的灰度图像的灰度值，从而根据所述目标图像中第二点云数据的反射率以及与所述目标图像对应的灰度图像的灰度值，确定所述投影三维空间投影到所述图像数据上的最优位置。In one embodiment, when the calibration device projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it can obtain The projected three-dimensional space is projected onto the target image obtained from the image data collected by the camera, and the reflectivity of the second point cloud data in the target image is determined, and the gray scale of the grayscale image corresponding to the target image is determined. Degree value, so as to determine the optimal projected three-dimensional space projected onto the image data according to the reflectivity of the second point cloud data in the target image and the gray value of the gray image corresponding to the target image position.
在一个实施例中，所述标定设备在根据所述目标图像中第二点云数据的反射率以及与所述目标图像对应的灰度图像的灰度值，确定所述投影三维空间投影到所述图像数据上的最优位置时，可以根据公式(2)确定所述投影三维空间投影到所述图像数据上的最优位置。In an embodiment, the calibration device determines that the projection three-dimensional space is projected to the target image according to the reflectivity of the second point cloud data in the target image and the gray value of the gray image corresponding to the target image. When the optimal position on the image data is described, the optimal position of the projection three-dimensional space projected on the image data can be determined according to formula (2).
Figure PCTCN2019098354-appb-000002
Figure PCTCN2019098354-appb-000002
其中，I p为图像上对应投影点的灰度值。 Among them, I p is the gray value of the corresponding projection point on the image.
在一个实施例中，所述标定设备将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，可以获取所述可移动平台在移动过程中的运动信息，并根据所述运动信息，确定所述第二点云数据的补偿信息，以及根据所述补偿信息对所述投影三维空间中的第二点云数据进行补偿，从而将补偿后的所述第二点云数据投影至所述相机采集的图像数据上，以获取所述投影三维空间投影到所述图像数据上的最优位置。在某些实施例中，所述运动信息包括位置、速度信息、加速度信息中的任意一种或多种。In one embodiment, when the calibration device projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it can obtain The movement information of the movable platform in the moving process, and according to the movement information, the compensation information of the second point cloud data is determined, and the second point cloud data in the projected three-dimensional space is determined according to the compensation information Compensation is performed, so that the compensated second point cloud data is projected onto the image data collected by the camera to obtain an optimal position of the projection three-dimensional space projected onto the image data. In some embodiments, the motion information includes any one or more of position, speed information, and acceleration information.
可见，通过这种实施方式，可以避免由于可移动平台的运动速度太快时，导致积累的点云数据出现模糊的情况，通过对点云数据进行补偿提高在可移动平台运动过程中采集的点云数据与图像数据的一致性。It can be seen that through this implementation method, it is possible to prevent the accumulated point cloud data from being blurred when the movable platform moves too fast, and the point cloud data can be compensated to improve the points collected during the movement of the movable platform. The consistency of cloud data and image data.
在一个实施例中，所述标定设备将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，可以获取所述可移动平台在预设时间范围内移动过程中的第二点云数据， 并将在所述预设时间范围内获取到的所述投影三维空间中的第二点云数据投影至所述相机采集的图像数据上，以获取所述投影三维空间投影到所述图像数据上的最优位置。In one embodiment, when the calibration device projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it can obtain The second point cloud data in the process of moving the movable platform within a preset time range, and the second point cloud data in the projection three-dimensional space acquired within the preset time range is projected to the camera On the collected image data to obtain the optimal position of the projection three-dimensional space projected onto the image data.
可见，通过降低点云的累积时间使用尽可能短时间的点数据的这种实施方式，可以提高在可移动平台运动过程中采集的点云数据与图像数据的一致性。It can be seen that by reducing the accumulation time of the point cloud and using the point data as short as possible, the consistency of the point cloud data and the image data collected during the movement of the movable platform can be improved.
可以理解，相机采集的图像数据并不限定为灰度图像，本实施例仅为示例性说明，在此不作限定。例如，也可以对相机采集的彩色图像数据进行处理。具体地，可以通过机器学习等算法先识别出场景中的具体物体，例如车道线、电线杆等，根据识别出的具体物体的反射率、亮度等物理信息，确定所述投影三维空间投影到所述图像数据上的最优位置，如此，可以减小误检概率，提高标定精度。It can be understood that the image data collected by the camera is not limited to grayscale images, and this embodiment is only an exemplary description and is not limited herein. For example, the color image data collected by the camera can also be processed. Specifically, algorithms such as machine learning can be used to first identify specific objects in the scene, such as lane lines, telephone poles, etc., and determine that the projected three-dimensional space is projected to all objects based on physical information such as the reflectance and brightness of the identified specific objects. According to the optimal position on the image data, the probability of false detection can be reduced and the calibration accuracy can be improved.
在一个实施例中，所述标定设备将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，可以获取所述可移动平台在移动过程中将所述投影三维空间投影至所述相机采集的图像数据上得到的多个目标图像，并将每个目标图像的数据进行比较，如果确定每个目标图像的数据一致，则可以确定所述目标图像的位置信息为所述投影三维空间投影到所述图像数据上的最优位置。In one embodiment, when the calibration device projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it can obtain During the movement of the movable platform, the projection three-dimensional space is projected onto multiple target images obtained by the image data collected by the camera, and the data of each target image is compared. If the data of each target image is determined If they are consistent, it can be determined that the position information of the target image is the optimal position projected onto the image data in the projected three-dimensional space.
在一个实施例中，所述标定设备在将每个目标图像的数据进行比较时，如果确定每个所述目标图像的数据不一致，则可以确定激光扫描装置的外参发生变化；进一步地，可以对所述激光扫描装置的外参进行更新。In one embodiment, when the calibration device compares the data of each target image, if it is determined that the data of each target image is inconsistent, it can be determined that the external parameters of the laser scanning device have changed; further, The external parameters of the laser scanning device are updated.
在一个实施例中，所述标定设备在将每个目标图像的数据进行比较时，如果确定每个所述目标图像的数据不一致，则可以触发预设的报警装置进行报警，以提示用户该激光扫描装置的外参发生变化，进一步地，还可以提示用户对激光扫描装置进行检查，或者自动对激光扫描装置进行检查，在此不作限定。In one embodiment, when the calibration device compares the data of each target image, if it is determined that the data of each target image is inconsistent, it can trigger a preset alarm device to give an alarm to remind the user of the laser The external parameters of the scanning device are changed, and further, the user may be prompted to check the laser scanning device or automatically check the laser scanning device, which is not limited here.
本发明实施例中，标定设备通过取激光扫描装置采集的可移动平台周围环境的第一点云数据以及相机采集的图像数据，并根据所述第一点云数据确定第二点云数据，以及将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间，当所述投影三维空间中的每个栅格区域满足预设条件时，将所述投影三维空间投影至所述图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置，从而实现了在没有特定标志物时对可移动平台周 围环境进行标定，提高了标定精度。In the embodiment of the present invention, the calibration device takes the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera, and determines the second point cloud data according to the first point cloud data, and Project the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space. When each grid area in the projected three-dimensional space meets a preset condition, the projected three-dimensional space Projecting onto the image data, and obtaining the optimal position of the projected three-dimensional space projected onto the image data, thereby achieving calibration of the surrounding environment of the movable platform when there is no specific marker, and improving the calibration accuracy.
下面结合附图对离线标定和在线标定的过程进行举例说明。The process of off-line calibration and on-line calibration will be illustrated below with reference to the drawings.
请参见图5，图5是本发明实施例提供的一种离线标定方法的流程示意图，如图5所示，在所述离线标定的过程中，通过激光雷达采集可移动平台周围环境的第一点云数据，通过相机采集图像数据，根据所述第一点云数据检测点云深度不连续点，并确定所述点云深度不连续点为第二点云数据，将所述第二点云数据投影到三维栅格空间，得到投影三维空间，将所述投影三维空间和已有数据比较相似性，如果相似，则丢弃该帧数据，如果不相似，则将所述投影三维空间的数据加入数据库，当确定所述数据库中的数据足够时，求取所述投影三维空间投影到所述图像数据上的最优位置。Please refer to FIG. 5, which is a schematic flowchart of an offline calibration method provided by an embodiment of the present invention. As shown in FIG. 5, in the offline calibration process, the first step of the surrounding environment of the movable platform is collected by lidar. Point cloud data, image data is collected by a camera, a point cloud depth discontinuity point is detected according to the first point cloud data, and the point cloud depth discontinuity point is determined to be the second point cloud data, and the second point cloud Project the data into the three-dimensional grid space to obtain the projected three-dimensional space, compare the projected three-dimensional space with the existing data, if it is similar, discard the frame data, if not, add the projected three-dimensional space data A database, when it is determined that the data in the database is sufficient, the optimal position of the projection three-dimensional space projected onto the image data is obtained.
请参见图6，图6是本发明实施例提供的一种在线标定方法的流程示意图，如图6所示，所述在线标定的过程包括所述离线标定的过程，此处对在线标定过程中包括的离线标定过程不再赘述，所述在线标定的过程与离线标定的过程的不同之处在于，在所述在线标定过程中，在求取到所述投影三维空间投影到所述图像数据上的最优位置后，可以对所述最优位置进行一致性检测。在某些实施例中，所述一致性检测包括：将所述最优位置的结果存储到结果队列中，并根据所述结果队列中存储多个最优位置，检测各最优位置是否一致，并输出检测结果，根据检测结果判断所述最优位置是否与图像数据一致，如果一致，则说明外参发生了变化，需要对最优位置进行更新，如果不一致，则说明结构有可能松动，不能完成标定。进一步地，当最优位置与图像数据不一致时，可以触发预设的报警装置进行报警，以提示用户该激光扫描装置的外参发生变化，或者提示用户对激光扫描装置进行检查，也可以自动对激光扫描装置进行检查，在此不作限定。Please refer to Figure 6. Figure 6 is a schematic flow chart of an online calibration method provided by an embodiment of the present invention. As shown in Figure 6, the online calibration process includes the offline calibration process. The included offline calibration process will not be repeated here. The difference between the online calibration process and the offline calibration process is that in the online calibration process, the projection three-dimensional space is obtained and projected onto the image data After the optimal position, the consistency check can be performed on the optimal position. In some embodiments, the consistency detection includes: storing the result of the optimal position in a result queue, and storing multiple optimal positions in the result queue to detect whether the optimal positions are consistent, And output the detection results, and judge whether the optimal position is consistent with the image data according to the detection results. If they are consistent, the external parameters have changed and the optimal position needs to be updated. If they are inconsistent, the structure may be loose and cannot Complete calibration. Further, when the optimal position is inconsistent with the image data, the preset alarm device can be triggered to give an alarm to remind the user that the external parameters of the laser scanning device have changed, or prompt the user to check the laser scanning device, or automatically The laser scanning device performs inspection, which is not limited here.
请参见图7，图7是本发明实施例提供的一种标定设备的结构示意图。具体的，所述标定设备包括：存储器701、处理器702。Please refer to FIG. 7, which is a schematic structural diagram of a calibration device provided by an embodiment of the present invention. Specifically, the calibration device includes: a memory 701 and a processor 702.
在一种实施例中，所述标定设备还包括数据接口703，所述数据接口703，用于传递标定设备和其他设备之间的数据信息。In an embodiment, the calibration device further includes a data interface 703, and the data interface 703 is used to transfer data information between the calibration device and other devices.
所述存储器701可以包括易失性存储器(volatile memory)；存储器701也可以包括非易失性存储器(non-volatile memory)；存储器701还可以包括上 述种类的存储器的组合。所述处理器702可以是中央处理器(central processing unit，CPU)。所述处理器702还可以进一步包括硬件芯片。上述硬件芯片可以是专用集成电路(application-specific integrated circuit，ASIC)，可编程逻辑器件(programmable logic device，PLD)或其组合。上述PLD可以是复杂可编程逻辑器件(complex programmable logic device，CPLD)，现场可编程逻辑门阵列(field-programmable gate array，FPGA)或其任意组合。The memory 701 may include a volatile memory (volatile memory); the memory 701 may also include a non-volatile memory (non-volatile memory); the memory 701 may also include a combination of the foregoing types of memories. The processor 702 may be a central processing unit (CPU). The processor 702 may further include a hardware chip. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The foregoing PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), or any combination thereof.
所述存储器701用于存储程序，所述处理器702可以调用存储器701中存储的程序，用于执行如下步骤：The memory 701 is used to store a program, and the processor 702 can call the program stored in the memory 701 to perform the following steps:
获取激光扫描装置采集的可移动平台周围环境的第一点云数据以及相机采集的图像数据；Acquiring the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera;
根据所述第一点云数据确定第二点云数据，所述第二点云数据用于指示无效的点云数据和/或不连续的点云数据；Determining second point cloud data according to the first point cloud data, where the second point cloud data is used to indicate invalid point cloud data and/or discontinuous point cloud data;
将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间；Projecting the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space;
当所述投影三维空间中的每个栅格区域满足预设条件时，将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置。When each grid area in the projected three-dimensional space meets a preset condition, project the projected three-dimensional space onto the image data collected by the camera, and obtain the projected three-dimensional space and project it onto the image data The best location.
进一步地，所述处理器702根据所述第一点云数据确定第二点云数据时，具体用于：Further, when the processor 702 determines the second point cloud data according to the first point cloud data, it is specifically configured to:
确定所述第一点云数据中相邻两个所述第一点云数据之间的距离；Determining the distance between two adjacent first point cloud data in the first point cloud data;
根据所述相邻两个所述第一点云数据之间的距离，确定不连续的第二点云数据。According to the distance between the two adjacent first point cloud data, the discontinuous second point cloud data is determined.
进一步地，所述处理器702根据所述相邻两个所述第一点云数据之间的距离，确定不连续的第二点云数据时，具体用于：Further, when the processor 702 determines the discontinuous second point cloud data according to the distance between the two adjacent first point cloud data, it is specifically configured to:
确定所述相邻两个所述第一点云数据之间的距离是否大于第一预设阈值；Determining whether the distance between the two adjacent first point cloud data is greater than a first preset threshold;
当确定出所述相邻两个所述第一点云数据之间的距离大于第一预设阈值时，确定所述相邻两个所述第一点云数据为不连续的第二点云数据。When it is determined that the distance between the two adjacent first point cloud data is greater than a first preset threshold, it is determined that the two adjacent first point cloud data are discontinuous second point clouds data.
进一步地，所述处理器702还用于：Further, the processor 702 is further configured to:
获取所述第一点云数据和原点之间的距离；Acquiring the distance between the first point cloud data and the origin;
根据所述第一点云数据和原点之间的距离以及所述相邻两个所述第一点 云数据之间的距离，确定所述不连续的第二点云数据。The discontinuous second point cloud data is determined according to the distance between the first point cloud data and the origin and the distance between the two adjacent first point cloud data.
进一步地，所述处理器702根据所述第一点云数据和原点之间的距离以及所述相邻两个所述第一点云数据之间的距离，确定所述不连续的第二点云数据时，具体用于：Further, the processor 702 determines the discontinuous second point according to the distance between the first point cloud data and the origin and the distance between the two adjacent first point cloud data When cloud data is used, it is specifically used for:
确定与原点之间的距离大于预设值的第一点云数据；Determine the first point cloud data whose distance from the origin is greater than a preset value;
从所述与原点之间的距离大于预设值的第一点云数据中，确定相邻两个所述第一点云数据之间的距离是否大于预设距离阈值；From the first point cloud data whose distance from the origin is greater than a preset value, determine whether the distance between two adjacent first point cloud data is greater than a preset distance threshold;
若大于所述预设距离阈值，则确定所述相邻两个所述第一点云数据为不连续的第二点云数据。If it is greater than the preset distance threshold, it is determined that the two adjacent first point cloud data are discontinuous second point cloud data.
进一步地，所述处理器702根据所述第一点云数据确定第二点云数据时，具体用于：Further, when the processor 702 determines the second point cloud data according to the first point cloud data, it is specifically configured to:
确定所述第一点云数据中是否存在深度信息；Determining whether depth information exists in the first point cloud data;
根据所述深度信息确定所述第一点云数据中为无效的所述第二点云数据。The invalid second point cloud data in the first point cloud data is determined according to the depth information.
进一步地，所述处理器702根据所述深度信息确定所述第二点云数据时，具体用于：Further, when the processor 702 determines the second point cloud data according to the depth information, it is specifically configured to:
从所述第一点云数据中确定不存在深度信息的所述第一点云数据为无效的所述第二点云数据。It is determined from the first point cloud data that the first point cloud data without depth information is the invalid second point cloud data.
进一步地，所述处理器702根据所述深度信息确定所述第二点云数据时，具体用于：Further, when the processor 702 determines the second point cloud data according to the depth information, it is specifically configured to:
获取所述第一点云数据的深度信息的变化值；Acquiring a change value of the depth information of the first point cloud data;
当所述第一点云数据的深度信息的变化值大于第二预设阈值时，确定所述大于第二预设阈值对应的所述第一点云数据为无效的所述第二点云数据。When the change value of the depth information of the first point cloud data is greater than a second preset threshold, it is determined that the first point cloud data corresponding to the greater than the second preset threshold is invalid second point cloud data .
进一步地，所述处理器702根据所述第一点云数据确定第二点云数据之前，还用于：Further, before determining the second point cloud data according to the first point cloud data, the processor 702 is further configured to:
将获取到的当前帧的第一点云数据与已经获取到的所述第一点云数据进行匹配；Matching the acquired first point cloud data of the current frame with the already acquired first point cloud data;
确定所述当前帧的第一点云数据的空间分布与所述已经获取到的所述第一点云数据的空间分布的相似度；Determining the degree of similarity between the spatial distribution of the first point cloud data of the current frame and the spatial distribution of the acquired first point cloud data;
如果所述相似度大于预设相似度阈值，则删除所述当前帧的第一点云数据；If the similarity is greater than the preset similarity threshold, delete the first point cloud data of the current frame;
如果所述相似度小于或等于所述预设相似度阈值，则确定将所述当前帧的第一点云数据加入已经获取到的所述第一点云数据。If the similarity is less than or equal to the preset similarity threshold, it is determined to add the first point cloud data of the current frame to the first point cloud data that has been acquired.
进一步地，所述处理器702将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间时，具体用于：Further, when the processor 702 projects the second point cloud data to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space, it is specifically used for:
确定所述激光扫描装置和相机之间的相对位置信息；Determining the relative position information between the laser scanning device and the camera;
根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间。According to the relative position information, the second point cloud data is projected to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space.
进一步地，所述处理器702根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间之前，还用于：Further, the processor 702 projects the second point cloud data to the three-dimensional grid space in the camera coordinate system according to the relative position information, and before obtaining the projected three-dimensional space, it is also used to:
确定所述第二点云数据与所述三维栅格空间中已存在的点云数据的空间分布相似度；Determining the spatial distribution similarity between the second point cloud data and the point cloud data that already exists in the three-dimensional grid space;
删除所述空间分布相似度大于预设相似度阈值的第二点云数据；Deleting the second point cloud data whose spatial distribution similarity is greater than a preset similarity threshold;
所述处理器702根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间时，具体用于：When the processor 702 projects the second point cloud data to the three-dimensional grid space in the camera coordinate system according to the relative position information, when obtaining the projected three-dimensional space, it is specifically used for:
根据所述相对位置信息将所述删除后的第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间。Projecting the deleted second point cloud data to the three-dimensional grid space in the camera coordinate system according to the relative position information to obtain the projected three-dimensional space.
进一步地，所述处理器702确定所述第二点云数据与所述三维栅格空间中已存在的点云数据的空间分布相似度时，具体用于：Further, when the processor 702 determines the spatial distribution similarity between the second point cloud data and the point cloud data that already exists in the three-dimensional grid space, it is specifically configured to:
确定所述第二点云数据的位置信息以及所述三维栅格空间中已存在的点云数据的位置信息；Determining the position information of the second point cloud data and the position information of the point cloud data that already exists in the three-dimensional grid space;
根据所述第二点云数据的位置信息与所述三维栅格空间中已存在的点云数据的位置信息，确定所述第二点云数据与所述三维栅格空间中已存在的点云数据的空间分布相似度。According to the position information of the second point cloud data and the position information of the point cloud data that already exists in the three-dimensional grid space, determine the second point cloud data and the point cloud that already exists in the three-dimensional grid space The spatial distribution similarity of the data.
进一步地，所述处理器702将所述第二点云数据投影至相机坐标系下的三维栅格空间之前，还用于：Further, before the processor 702 projects the second point cloud data to the three-dimensional grid space in the camera coordinate system, it is also used to:
确定所述相机的视角是否小于所述激光扫描装置的视角；Determining whether the angle of view of the camera is smaller than the angle of view of the laser scanning device;
当确定所述相机的视角小于所述激光扫描装置的视角时，执行所述将所述第二点云数据投影至相机坐标系下的三维栅格空间的步骤。When it is determined that the angle of view of the camera is smaller than the angle of view of the laser scanning device, the step of projecting the second point cloud data to the three-dimensional grid space in the camera coordinate system is performed.
进一步地，所述满足预设条件，包括：Further, the meeting the preset condition includes:
所述投影三维空间中的每个栅格区域中的第二点云数据的数量大于预设 数量阈值。The quantity of the second point cloud data in each grid area in the projected three-dimensional space is greater than a preset quantity threshold.
进一步地，所述处理器702将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：Further, when the processor 702 projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it is specifically used for:
根据所述相机采集的图像数据，确定与所述图像数据对应的梯度图像；Determining a gradient image corresponding to the image data according to the image data collected by the camera;
将所述投影三维空间的第二点云数据投影至所述梯度图像；Projecting the second point cloud data of the projected three-dimensional space onto the gradient image;
当确定所述投影三维空间的第二点云数据投影至所述梯度图像，所述投影三维空间的第二点云数据与所述梯度图像完全融合时，确定所述投影三维空间投影到所述图像数据上的最优位置。When it is determined that the second point cloud data of the projected three-dimensional space is projected to the gradient image, and the second point cloud data of the projected three-dimensional space is completely fused with the gradient image, it is determined that the projected three-dimensional space is projected to the gradient image. The optimal position on the image data.
进一步地，所述处理器702根据所述相机采集的图像数据，确定与所述图像数据对应的梯度图像时，具体用于：Further, when the processor 702 determines the gradient image corresponding to the image data according to the image data collected by the camera, it is specifically configured to:
根据所述相机采集的图像数据，确定与所述图像数据对应的灰度图像；Determining a grayscale image corresponding to the image data according to the image data collected by the camera;
从所述灰度图像中提取梯度信息和/或边缘信息；Extracting gradient information and/or edge information from the grayscale image;
根据所述梯度信息和/或边缘信息，确定所述梯度图像。The gradient image is determined according to the gradient information and/or edge information.
进一步地，所述处理器702将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：Further, when the processor 702 projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it is specifically used for:
获取将所述投影三维空间投影至所述相机采集的图像数据上得到的目标图像；Acquiring a target image obtained by projecting the projected three-dimensional space onto the image data collected by the camera;
确定所述目标图像中第二点云数据的反射率；Determining the reflectivity of the second point cloud data in the target image;
确定与所述目标图像对应的灰度图像的灰度值；Determining the gray value of the gray image corresponding to the target image;
根据所述目标图像中第二点云数据的反射率以及与所述目标图像对应的灰度图像的灰度值，确定所述投影三维空间投影到所述图像数据上的最优位置。According to the reflectance of the second point cloud data in the target image and the gray value of the gray image corresponding to the target image, the optimal position of the projection three-dimensional space projected onto the image data is determined.
进一步地，所述处理器702将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：Further, when the processor 702 projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it is specifically used for:
获取所述可移动平台在移动过程中的运动信息；Acquiring movement information of the movable platform during the movement;
根据所述运动信息，确定所述第二点云数据的补偿信息；Determine the compensation information of the second point cloud data according to the motion information;
根据所述补偿信息对所述投影三维空间中的第二点云数据进行补偿；Compensate the second point cloud data in the projected three-dimensional space according to the compensation information;
将补偿后的所述第二点云数据投影至所述相机采集的图像数据上，以获取所述投影三维空间投影到所述图像数据上的最优位置。Projecting the compensated second point cloud data onto the image data collected by the camera to obtain the optimal position of the projected three-dimensional space projected onto the image data.
进一步地，所述运动信息包括位置信息、速度信息、加速度信息中的任意一种或多种。Further, the motion information includes any one or more of position information, speed information, and acceleration information.
进一步地，所述处理器702将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：Further, when the processor 702 projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it is specifically used for:
获取所述可移动平台在预设时间范围内移动过程中的第二点云数据；Acquiring second point cloud data during the movement of the movable platform within a preset time range;
将在所述预设时间范围内获取到的所述投影三维空间中的第二点云数据投影至所述相机采集的图像数据上，以获取所述投影三维空间投影到所述图像数据上的最优位置。Projecting the second point cloud data in the projected three-dimensional space acquired within the preset time range onto the image data collected by the camera to obtain the projected three-dimensional space projected on the image data Optimal location.
进一步地，所述处理器702将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：Further, when the processor 702 projects the projected three-dimensional space onto the image data collected by the camera, and obtains the optimal position of the projected three-dimensional space onto the image data, it is specifically used for:
获取所述可移动平台在移动过程中将所述投影三维空间投影至所述相机采集的图像数据上得到的多个目标图像；Acquiring a plurality of target images obtained by projecting the projected three-dimensional space onto the image data collected by the camera during the movement of the movable platform;
将每个目标图像的数据进行比较；Compare the data of each target image;
如果确定每个目标图像的数据一致，则确定所述目标图像的位置信息为所述投影三维空间投影到所述图像数据上的最优位置。If it is determined that the data of each target image is consistent, it is determined that the position information of the target image is the optimal position of the projection three-dimensional space projected onto the image data.
进一步地，所述处理器702还用于：Further, the processor 702 is further configured to:
如果确定每个所述目标图像的数据不一致，则确定激光扫描装置的外参发生变化，并对所述激光扫描装置的外参进行更新。If it is determined that the data of each target image is inconsistent, it is determined that the external parameters of the laser scanning device have changed, and the external parameters of the laser scanning device are updated.
进一步地，所述处理器702所述方法还用于：Further, the method described by the processor 702 is further configured to:
如果确定每个所述目标图像的数据不一致，则触发预设的报警装置进行报警，以提示用户对激光扫描装置进行检查。If it is determined that the data of each target image is inconsistent, a preset alarm device is triggered to give an alarm to prompt the user to check the laser scanning device.
进一步地，所述激光扫描装置包括激光雷达、毫米波雷达、超声波雷达中的任意一种或多种。Further, the laser scanning device includes any one or more of laser radar, millimeter wave radar, and ultrasonic radar.
本发明实施例中，标定设备通过获取激光扫描装置采集的可移动平台周围环境的第一点云数据以及相机采集的图像数据，并根据所述第一点云数据确定第二点云数据，以及将所述第二点云数据投影至相机坐标系下的三维栅格空 间，得到投影三维空间，当所述投影三维空间中的每个栅格区域满足预设条件时，将所述投影三维空间投影至所述图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置，从而实现了在没有特定标志物时对可移动平台周围环境进行标定，提高了标定精度。In the embodiment of the present invention, the calibration device obtains the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera, and determines the second point cloud data according to the first point cloud data, and Project the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space. When each grid area in the projected three-dimensional space meets a preset condition, the projected three-dimensional space Projecting onto the image data, and obtaining the optimal position of the projected three-dimensional space projected onto the image data, thereby achieving calibration of the surrounding environment of the movable platform when there is no specific marker, and improving the calibration accuracy.
本发明实施例还提供了一种可移动平台，所述可移动平台包括：机身；配置在机身上的动力系统，用于为可移动平台提供移动的动力；以及上述标定设备。本发明实施例中，可移动平台通过获取激光扫描装置采集的可移动平台周围环境的第一点云数据以及相机采集的图像数据，并根据所述第一点云数据确定第二点云数据，以及将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间，当所述投影三维空间中的每个栅格区域满足预设条件时，将所述投影三维空间投影至所述图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置，从而实现了在没有特定标志物时对可移动平台周围环境进行标定，提高了标定精度。The embodiment of the present invention also provides a movable platform, the movable platform includes: a fuselage; a power system configured on the fuselage for providing mobile power for the movable platform; and the above-mentioned calibration equipment. In the embodiment of the present invention, the movable platform obtains the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera, and determines the second point cloud data according to the first point cloud data, And projecting the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space. When each grid area in the projected three-dimensional space meets a preset condition, the projected three-dimensional Space projection onto the image data, and obtain the optimal position of the projected three-dimensional space projected onto the image data, so as to realize the calibration of the surrounding environment of the movable platform when there is no specific marker, and improve the calibration accuracy .
本发明的实施例还提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行时实现本发明图2所对应实施例中描述的方法，也可实现图7所述本发明所对应实施例的设备，在此不再赘述。The embodiment of the present invention also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method described in the embodiment corresponding to FIG. 2 of the present invention , The device corresponding to the embodiment of the present invention described in FIG. 7 can also be implemented, which will not be repeated here.
所述计算机可读存储介质可以是前述任一实施例所述的设备的内部存储单元，例如设备的硬盘或内存。所述计算机可读存储介质也可以是所述设备的外部存储设备，例如所述设备上配备的插接式硬盘，智能存储卡(Smart Media Card,SMC)，安全数字(Secure Digital,SD)卡，闪存卡(Flash Card)等。进一步地，所述计算机可读存储介质还可以既包括所述设备的内部存储单元也包括外部存储设备。所述计算机可读存储介质用于存储所述计算机程序以及所述终端所需的其他程序和数据。所述计算机可读存储介质还可以用于暂时地存储已经输出或者将要输出的数据。The computer-readable storage medium may be an internal storage unit of the device described in any of the foregoing embodiments, such as a hard disk or memory of the device. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk equipped on the device, a Smart Media Card (SMC), or a Secure Digital (SD) card , Flash Card, etc. Further, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.
以上所揭露的仅为本发明部分实施例而已，当然不能以此来限定本发明之权利范围，因此依本发明权利要求所作的等同变化，仍属本发明所涵盖的范围。The above-disclosed are only some embodiments of the present invention, which of course cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention still fall within the scope of the present invention.

Claims (74)

  1. 一种标定方法，其特征在于，应用于可移动平台，所述可移动平台包括激光扫描装置和相机，所述方法包括：A calibration method, characterized by being applied to a movable platform, the movable platform including a laser scanning device and a camera, and the method includes:
    获取所述激光扫描装置采集的所述可移动平台周围环境的第一点云数据以及所述相机采集的图像数据；Acquiring first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and image data collected by the camera;
    根据所述第一点云数据确定第二点云数据，所述第二点云数据用于指示无效的点云数据和/或不连续的点云数据；Determining second point cloud data according to the first point cloud data, where the second point cloud data is used to indicate invalid point cloud data and/or discontinuous point cloud data;
    将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间；Projecting the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space;
    当所述投影三维空间中的每个栅格区域满足预设条件时，将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置。When each grid area in the projected three-dimensional space meets a preset condition, project the projected three-dimensional space onto the image data collected by the camera, and obtain the projected three-dimensional space and project it onto the image data The best location.
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述第一点云数据确定第二点云数据，包括：The method according to claim 1, wherein the determining second point cloud data according to the first point cloud data comprises:
    确定所述第一点云数据中相邻两个所述第一点云数据之间的距离；Determining the distance between two adjacent first point cloud data in the first point cloud data;
    根据所述相邻两个所述第一点云数据之间的距离，确定不连续的第二点云数据。According to the distance between the two adjacent first point cloud data, the discontinuous second point cloud data is determined.
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述相邻两个所述第一点云数据之间的距离，确定不连续的第二点云数据，包括：The method according to claim 2, wherein the determining the discontinuous second point cloud data according to the distance between the two adjacent first point cloud data comprises:
    确定所述相邻两个所述第一点云数据之间的距离是否大于第一预设阈值；Determining whether the distance between the two adjacent first point cloud data is greater than a first preset threshold;
    当确定出所述相邻两个所述第一点云数据之间的距离大于第一预设阈值时，确定所述相邻两个所述第一点云数据为不连续的第二点云数据。When it is determined that the distance between the two adjacent first point cloud data is greater than a first preset threshold, it is determined that the two adjacent first point cloud data are discontinuous second point clouds data.
  4. 根据权利要求2所述的方法，其特征在于，所述方法还包括：The method of claim 2, wherein the method further comprises:
    获取所述第一点云数据和原点之间的距离；Acquiring the distance between the first point cloud data and the origin;
    根据所述第一点云数据和原点之间的距离以及所述相邻两个所述第一点云数据之间的距离，确定所述不连续的第二点云数据。Determine the discontinuous second point cloud data according to the distance between the first point cloud data and the origin and the distance between the two adjacent first point cloud data.
  5. 根据权利要求4所述的方法，其特征在于，所述根据所述第一点云数据和原点之间的距离以及所述相邻两个所述第一点云数据之间的距离，确定所述不连续的第二点云数据，包括：The method according to claim 4, characterized in that the determining the distance between the first point cloud data and the origin and the distance between the two adjacent first point cloud data The discontinuous second point cloud data includes:
    确定与原点之间的距离大于预设值的第一点云数据；Determine the first point cloud data whose distance from the origin is greater than a preset value;
    从所述与原点之间的距离大于预设值的第一点云数据中，确定相邻两个所述第一点云数据之间的距离是否大于预设距离阈值；From the first point cloud data whose distance from the origin is greater than a preset value, determine whether the distance between two adjacent first point cloud data is greater than a preset distance threshold;
    若大于所述预设距离阈值，则确定所述相邻两个所述第一点云数据为不连续的第二点云数据。If it is greater than the preset distance threshold, it is determined that the two adjacent first point cloud data are discontinuous second point cloud data.
  6. 根据权利要求1所述的方法，其特征在于，所述根据所述第一点云数据确定第二点云数据，包括：The method according to claim 1, wherein the determining second point cloud data according to the first point cloud data comprises:
    确定所述第一点云数据中是否存在深度信息；Determining whether depth information exists in the first point cloud data;
    根据所述深度信息确定所述第一点云数据中为无效的所述第二点云数据。The invalid second point cloud data in the first point cloud data is determined according to the depth information.
  7. 根据权利要求6所述的方法，其特征在于，所述根据所述深度信息确定所述第二点云数据，包括：The method according to claim 6, wherein the determining the second point cloud data according to the depth information comprises:
    从所述第一点云数据中确定不存在深度信息的所述第一点云数据为无效的所述第二点云数据。It is determined from the first point cloud data that the first point cloud data without depth information is the invalid second point cloud data.
  8. 根据权利要求6所述的方法，其特征在于，所述根据所述深度信息确定所述第二点云数据，包括：The method according to claim 6, wherein the determining the second point cloud data according to the depth information comprises:
    获取所述第一点云数据的深度信息的变化值；Acquiring a change value of the depth information of the first point cloud data;
    当所述第一点云数据的深度信息的变化值大于第二预设阈值时，确定所述大于第二预设阈值对应的所述第一点云数据为无效的所述第二点云数据。When the change value of the depth information of the first point cloud data is greater than a second preset threshold, it is determined that the first point cloud data corresponding to the greater than the second preset threshold is invalid second point cloud data .
  9. 根据权利要求1所述的方法，其特征在于，所述根据所述第一点云数据确定第二点云数据之前，还包括：The method according to claim 1, wherein before determining the second point cloud data according to the first point cloud data, the method further comprises:
    将获取到的当前帧的第一点云数据与已经获取到的所述第一点云数据进行匹配；Matching the acquired first point cloud data of the current frame with the already acquired first point cloud data;
    确定所述当前帧的第一点云数据的空间分布与所述已经获取到的所述第一点云数据的空间分布的相似度；Determining the degree of similarity between the spatial distribution of the first point cloud data of the current frame and the spatial distribution of the acquired first point cloud data;
    如果所述相似度大于预设相似度阈值，则删除所述当前帧的第一点云数据；If the similarity is greater than the preset similarity threshold, delete the first point cloud data of the current frame;
    如果所述相似度小于或等于所述预设相似度阈值，则确定将所述当前帧的第一点云数据加入已经获取到的所述第一点云数据。If the similarity is less than or equal to the preset similarity threshold, it is determined to add the first point cloud data of the current frame to the first point cloud data that has been acquired.
  10. 根据权利要求1所述的方法，其特征在于，所述将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间，包括：The method according to claim 1, wherein the projecting the second point cloud data to a three-dimensional raster space in a camera coordinate system to obtain a projected three-dimensional space comprises:
    确定所述激光扫描装置和相机之间的相对位置信息；Determining the relative position information between the laser scanning device and the camera;
    根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间。According to the relative position information, the second point cloud data is projected to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space.
  11. 根据权利要求10所述的方法，其特征在于，所述根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间之前，还包括：The method according to claim 10, characterized in that, before projecting the second point cloud data to a three-dimensional grid space in a camera coordinate system according to the relative position information, before obtaining the projected three-dimensional space, the method further comprises:
    确定所述第二点云数据与所述三维栅格空间中已存在的点云数据的空间分布相似度；Determining the spatial distribution similarity between the second point cloud data and the point cloud data that already exists in the three-dimensional grid space;
    删除所述空间分布相似度大于预设相似度阈值的第二点云数据；Deleting the second point cloud data whose spatial distribution similarity is greater than a preset similarity threshold;
    所述根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间，包括：The projecting the second point cloud data to the three-dimensional grid space in the camera coordinate system according to the relative position information to obtain the projected three-dimensional space includes:
    根据所述相对位置信息将所述删除后的第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间。Projecting the deleted second point cloud data to the three-dimensional grid space in the camera coordinate system according to the relative position information to obtain the projected three-dimensional space.
  12. 根据权利要求11所述的方法，其特征在于，所述确定所述第二点云数据与所述三维栅格空间中已存在的点云数据的空间分布相似度，包括：The method according to claim 11, wherein the determining the spatial distribution similarity between the second point cloud data and the point cloud data that already exists in the three-dimensional grid space comprises:
    确定所述第二点云数据的位置信息以及所述三维栅格空间中已存在的点云数据的位置信息；Determining the position information of the second point cloud data and the position information of the point cloud data that already exists in the three-dimensional grid space;
    根据所述第二点云数据的位置信息与所述三维栅格空间中已存在的点云数据的位置信息，确定所述第二点云数据与所述三维栅格空间中已存在的点云 数据的空间分布相似度。According to the position information of the second point cloud data and the position information of the point cloud data that already exists in the three-dimensional grid space, determine the second point cloud data and the point cloud that already exists in the three-dimensional grid space The spatial distribution similarity of the data.
  13. 根据权利要求10所述的方法，其特征在于，所述将所述第二点云数据投影至相机坐标系下的三维栅格空间之前，还包括：The method according to claim 10, wherein before the projecting the second point cloud data to a three-dimensional grid space in a camera coordinate system, the method further comprises:
    确定所述相机的视角是否小于所述激光扫描装置的视角；Determining whether the angle of view of the camera is smaller than the angle of view of the laser scanning device;
    当确定所述相机的视角小于所述激光扫描装置的视角时，执行所述将所述第二点云数据投影至相机坐标系下的三维栅格空间的步骤。When it is determined that the angle of view of the camera is smaller than the angle of view of the laser scanning device, the step of projecting the second point cloud data to the three-dimensional grid space in the camera coordinate system is performed.
  14. 根据权利要求1所述的方法，其特征在于，所述满足预设条件，包括：The method according to claim 1, wherein the satisfying a preset condition comprises:
    所述投影三维空间中的每个栅格区域中的第二点云数据的数量大于预设数量阈值。The quantity of the second point cloud data in each grid area in the projection three-dimensional space is greater than a preset quantity threshold.
  15. 根据权利要求1所述的方法，其特征在于，所述将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置，包括：The method according to claim 1, wherein the projecting the projected three-dimensional space onto the image data collected by the camera, and obtaining the optimal position of the projected three-dimensional space onto the image data ,include:
    根据所述相机采集的图像数据，确定与所述图像数据对应的梯度图像；Determining a gradient image corresponding to the image data according to the image data collected by the camera;
    将所述投影三维空间的第二点云数据投影至所述梯度图像；Projecting the second point cloud data of the projected three-dimensional space onto the gradient image;
    当确定所述投影三维空间的第二点云数据投影至所述梯度图像，所述投影三维空间的第二点云数据与所述梯度图像完全融合时，确定所述投影三维空间投影到所述图像数据上的最优位置。When it is determined that the second point cloud data of the projected three-dimensional space is projected to the gradient image, and the second point cloud data of the projected three-dimensional space is completely fused with the gradient image, it is determined that the projected three-dimensional space is projected to the gradient image. The optimal position on the image data.
  16. 根据权利要求15所述的方法，其特征在于，所述根据所述相机采集的图像数据，确定与所述图像数据对应的梯度图像，包括：The method according to claim 15, wherein the determining a gradient image corresponding to the image data according to the image data collected by the camera comprises:
    根据所述相机采集的图像数据，确定与所述图像数据对应的灰度图像；Determining a grayscale image corresponding to the image data according to the image data collected by the camera;
    从所述灰度图像中提取梯度信息和/或边缘信息；Extracting gradient information and/or edge information from the grayscale image;
    根据所述梯度信息和/或边缘信息，确定所述梯度图像。The gradient image is determined according to the gradient information and/or edge information.
  17. 根据权利要求1所述的方法，其特征在于，所述将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置，包括：The method according to claim 1, wherein the projecting the projected three-dimensional space onto the image data collected by the camera, and obtaining the optimal position of the projected three-dimensional space onto the image data ,include:
    获取将所述投影三维空间投影至所述相机采集的图像数据上得到的目标图像；Acquiring a target image obtained by projecting the projected three-dimensional space onto the image data collected by the camera;
    确定所述目标图像中第二点云数据的反射率；Determining the reflectivity of the second point cloud data in the target image;
    确定与所述目标图像对应的灰度图像的灰度值；Determining the gray value of the gray image corresponding to the target image;
    根据所述目标图像中第二点云数据的反射率以及与所述目标图像对应的灰度图像的灰度值，确定所述投影三维空间投影到所述图像数据上的最优位置。According to the reflectance of the second point cloud data in the target image and the gray value of the gray image corresponding to the target image, the optimal position of the projection three-dimensional space projected onto the image data is determined.
  18. 根据权利要求1所述的方法，其特征在于，所述将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置，包括：The method according to claim 1, wherein the projecting the projected three-dimensional space onto the image data collected by the camera, and obtaining the optimal position of the projected three-dimensional space onto the image data ,include:
    获取所述可移动平台在移动过程中的运动信息；Acquiring movement information of the movable platform during the movement;
    根据所述运动信息，确定所述第二点云数据的补偿信息；Determine the compensation information of the second point cloud data according to the motion information;
    根据所述补偿信息对所述投影三维空间中的第二点云数据进行补偿；Compensate the second point cloud data in the projected three-dimensional space according to the compensation information;
    将补偿后的所述第二点云数据投影至所述相机采集的图像数据上，以获取所述投影三维空间投影到所述图像数据上的最优位置。Projecting the compensated second point cloud data onto the image data collected by the camera to obtain the optimal position of the projected three-dimensional space projected onto the image data.
  19. 根据权利要求18所述的方法，其特征在于，The method of claim 18, wherein:
    所述运动信息包括位置信息、速度信息、加速度信息中的任意一种或多种。The motion information includes any one or more of position information, speed information, and acceleration information.
  20. 根据权利要求1所述的方法，其特征在于，所述将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置，包括：The method according to claim 1, wherein the projection of the three-dimensional projection space onto the image data collected by the camera, and obtaining the optimal position of the projection three-dimensional space onto the image data ,include:
    获取所述可移动平台在预设时间范围内移动过程中的第二点云数据；Acquiring second point cloud data during the movement of the movable platform within a preset time range;
    将在所述预设时间范围内获取到的所述投影三维空间中的第二点云数据投影至所述相机采集的图像数据上，以获取所述投影三维空间投影到所述图像数据上的最优位置。Projecting the second point cloud data in the projected three-dimensional space acquired within the preset time range onto the image data collected by the camera to obtain the projected three-dimensional space projected on the image data Optimal location.
  21. 根据权利要求18或20所述的方法，其特征在于，所述将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所 述图像数据上的最优位置，包括：The method according to claim 18 or 20, wherein the projecting the projected three-dimensional space onto the image data collected by the camera, and obtaining the most of the projected three-dimensional space onto the image data Preferred location, including:
    获取所述可移动平台在移动过程中将所述投影三维空间投影至所述相机采集的图像数据上得到的多个目标图像；Acquiring a plurality of target images obtained by projecting the projected three-dimensional space onto the image data collected by the camera during the movement of the movable platform;
    将每个目标图像的数据进行比较；Compare the data of each target image;
    如果确定每个目标图像的数据一致，则确定所述目标图像的位置信息为所述投影三维空间投影到所述图像数据上的最优位置。If it is determined that the data of each target image is consistent, it is determined that the position information of the target image is the optimal position of the projection three-dimensional space projected onto the image data.
  22. 根据权利要求21所述的方法，其特征在于，所述方法还包括：The method of claim 21, wherein the method further comprises:
    如果确定每个所述目标图像的数据不一致，则确定激光扫描装置的外参发生变化，并对所述激光扫描装置的外参进行更新。If it is determined that the data of each target image is inconsistent, it is determined that the external parameters of the laser scanning device have changed, and the external parameters of the laser scanning device are updated.
  23. 根据权利要求21所述的方法，其特征在于，所述方法还包括：The method of claim 21, wherein the method further comprises:
    如果确定每个所述目标图像的数据不一致，则触发预设的报警装置进行报警，以提示用户对激光扫描装置进行检查。If it is determined that the data of each target image is inconsistent, a preset alarm device is triggered to give an alarm to prompt the user to check the laser scanning device.
  24. 根据权利要求1所述的方法，其特征在于，The method according to claim 1, wherein:
    所述激光扫描装置包括激光雷达、毫米波雷达、超声波雷达中的任意一种或多种。The laser scanning device includes any one or more of laser radar, millimeter wave radar, and ultrasonic radar.
  25. 一种标定设备，其特征在于，应用于可移动平台，所述可移动平台包括激光扫描装置和相机，所述设备包括存储器和处理器；A calibration equipment, characterized in that it is applied to a movable platform, the movable platform includes a laser scanning device and a camera, and the equipment includes a memory and a processor;
    所述存储器，用于存储程序；The memory is used to store programs;
    所述处理器，用于调用所述程序，当所述程序被执行时，用于执行以下操作:The processor is used to call the program, and when the program is executed, it is used to perform the following operations:
    获取所述激光扫描装置采集的所述可移动平台周围环境的第一点云数据以及所述相机采集的图像数据；Acquiring first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and image data collected by the camera;
    根据所述第一点云数据确定第二点云数据，所述第二点云数据用于指示无效的点云数据和/或不连续的点云数据；Determining second point cloud data according to the first point cloud data, where the second point cloud data is used to indicate invalid point cloud data and/or discontinuous point cloud data;
    将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间；Projecting the second point cloud data to a three-dimensional grid space in the camera coordinate system to obtain a projected three-dimensional space;
    当所述投影三维空间中的每个栅格区域满足预设条件时，将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置。When each grid area in the projected three-dimensional space meets a preset condition, project the projected three-dimensional space onto the image data collected by the camera, and obtain the projected three-dimensional space and project it onto the image data The best location.
  26. 根据权利要求25所述的设备，其特征在于，所述处理器根据所述第一点云数据确定第二点云数据时，具体用于：The device according to claim 25, wherein when the processor determines the second point cloud data according to the first point cloud data, it is specifically configured to:
    确定所述第一点云数据中相邻两个所述第一点云数据之间的距离；Determining the distance between two adjacent first point cloud data in the first point cloud data;
    根据所述相邻两个所述第一点云数据之间的距离，确定不连续的第二点云数据。According to the distance between the two adjacent first point cloud data, the discontinuous second point cloud data is determined.
  27. 根据权利要求26所述的设备，其特征在于，所述处理器根据所述相邻两个所述第一点云数据之间的距离，确定不连续的第二点云数据时，具体用于：The device according to claim 26, wherein when the processor determines the discontinuous second point cloud data according to the distance between the two adjacent first point cloud data, it is specifically configured to :
    确定所述相邻两个所述第一点云数据之间的距离是否大于第一预设阈值；Determining whether the distance between the two adjacent first point cloud data is greater than a first preset threshold;
    当确定出所述相邻两个所述第一点云数据之间的距离大于第一预设阈值时，确定所述相邻两个所述第一点云数据为不连续的第二点云数据。When it is determined that the distance between the two adjacent first point cloud data is greater than a first preset threshold, it is determined that the two adjacent first point cloud data are discontinuous second point clouds data.
  28. 根据权利要求26所述的设备，其特征在于，所述处理器还用于：The device according to claim 26, wherein the processor is further configured to:
    获取所述第一点云数据和原点之间的距离；Acquiring the distance between the first point cloud data and the origin;
    根据所述第一点云数据和原点之间的距离以及所述相邻两个所述第一点云数据之间的距离，确定所述不连续的第二点云数据。Determine the discontinuous second point cloud data according to the distance between the first point cloud data and the origin and the distance between the two adjacent first point cloud data.
  29. 根据权利要求28所述的设备，其特征在于，所述处理器根据所述第一点云数据和原点之间的距离以及所述相邻两个所述第一点云数据之间的距离，确定所述不连续的第二点云数据时，具体用于：The device according to claim 28, wherein the processor is based on the distance between the first point cloud data and the origin and the distance between the two adjacent first point cloud data, When determining the discontinuous second point cloud data, it is specifically used for:
    确定与原点之间的距离大于预设值的第一点云数据；Determine the first point cloud data whose distance from the origin is greater than a preset value;
    从所述与原点之间的距离大于预设值的第一点云数据中，确定相邻两个所述第一点云数据之间的距离是否大于预设距离阈值；From the first point cloud data whose distance from the origin is greater than a preset value, determine whether the distance between two adjacent first point cloud data is greater than a preset distance threshold;
    若大于所述预设距离阈值，则确定所述相邻两个所述第一点云数据为不连续的第二点云数据。If it is greater than the preset distance threshold, it is determined that the two adjacent first point cloud data are discontinuous second point cloud data.
  30. 根据权利要求25所述的设备，其特征在于，所述处理器根据所述第一点云数据确定第二点云数据时，具体用于：The device according to claim 25, wherein when the processor determines the second point cloud data according to the first point cloud data, it is specifically configured to:
    确定所述第一点云数据中是否存在深度信息；Determining whether depth information exists in the first point cloud data;
    根据所述深度信息确定所述第一点云数据中为无效的所述第二点云数据。The invalid second point cloud data in the first point cloud data is determined according to the depth information.
  31. 根据权利要求30所述的设备，其特征在于，所述处理器根据所述深度信息确定所述第二点云数据时，具体用于：The device according to claim 30, wherein when the processor determines the second point cloud data according to the depth information, it is specifically configured to:
    从所述第一点云数据中确定不存在深度信息的所述第一点云数据为无效的所述第二点云数据。It is determined from the first point cloud data that the first point cloud data without depth information is the invalid second point cloud data.
  32. 根据权利要求30所述的设备，其特征在于，所述处理器根据所述深度信息确定所述第二点云数据时，具体用于：The device according to claim 30, wherein when the processor determines the second point cloud data according to the depth information, it is specifically configured to:
    获取所述第一点云数据的深度信息的变化值；Acquiring a change value of the depth information of the first point cloud data;
    当所述第一点云数据的深度信息的变化值大于第二预设阈值时，确定所述大于第二预设阈值对应的所述第一点云数据为无效的所述第二点云数据。When the change value of the depth information of the first point cloud data is greater than a second preset threshold, it is determined that the first point cloud data corresponding to the greater than the second preset threshold is invalid second point cloud data .
  33. 根据权利要求25所述的设备，其特征在于，所述处理器根据所述第一点云数据确定第二点云数据之前，还用于：The device according to claim 25, wherein the processor is further configured to: before determining the second point cloud data according to the first point cloud data:
    将获取到的当前帧的第一点云数据与已经获取到的所述第一点云数据进行匹配；Matching the acquired first point cloud data of the current frame with the already acquired first point cloud data;
    确定所述当前帧的第一点云数据的空间分布与所述已经获取到的所述第一点云数据的空间分布的相似度；Determining the degree of similarity between the spatial distribution of the first point cloud data of the current frame and the spatial distribution of the acquired first point cloud data;
    如果所述相似度大于预设相似度阈值，则删除所述当前帧的第一点云数据；If the similarity is greater than the preset similarity threshold, delete the first point cloud data of the current frame;
    如果所述相似度小于或等于所述预设相似度阈值，则确定将所述当前帧的第一点云数据加入已经获取到的所述第一点云数据。If the similarity is less than or equal to the preset similarity threshold, it is determined to add the first point cloud data of the current frame to the first point cloud data that has been acquired.
  34. 根据权利要求25所述的设备，其特征在于，所述处理器将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间时，具体用 于：The device according to claim 25, wherein the processor projects the second point cloud data to a three-dimensional grid space in a camera coordinate system, and when obtaining a projected three-dimensional space, it is specifically used for:
    确定所述激光扫描装置和相机之间的相对位置信息；Determining the relative position information between the laser scanning device and the camera;
    根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间。According to the relative position information, the second point cloud data is projected to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space.
  35. 根据权利要求34所述的设备，其特征在于，所述处理器根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间之前，还用于：The device according to claim 34, wherein the processor projects the second point cloud data to a three-dimensional grid space in a camera coordinate system according to the relative position information, and before obtaining the projected three-dimensional space, Used for:
    确定所述第二点云数据与所述三维栅格空间中已存在的点云数据的空间分布相似度；Determining the spatial distribution similarity between the second point cloud data and the point cloud data that already exists in the three-dimensional grid space;
    删除所述空间分布相似度大于预设相似度阈值的第二点云数据；Deleting the second point cloud data whose spatial distribution similarity is greater than a preset similarity threshold;
    所述处理器根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间时，具体用于：When the processor projects the second point cloud data to the three-dimensional grid space in the camera coordinate system according to the relative position information, when obtaining the projected three-dimensional space, it is specifically used for:
    根据所述相对位置信息将所述删除后的第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间。Projecting the deleted second point cloud data to the three-dimensional grid space in the camera coordinate system according to the relative position information to obtain the projected three-dimensional space.
  36. 根据权利要求35所述的设备，其特征在于，所述处理器确定所述第二点云数据与所述三维栅格空间中已存在的点云数据的空间分布相似度时，具体用于：The device according to claim 35, wherein when the processor determines the spatial distribution similarity between the second point cloud data and the point cloud data that already exists in the three-dimensional grid space, it is specifically configured to:
    确定所述第二点云数据的位置信息以及所述三维栅格空间中已存在的点云数据的位置信息；Determining the position information of the second point cloud data and the position information of the point cloud data that already exists in the three-dimensional grid space;
    根据所述第二点云数据的位置信息与所述三维栅格空间中已存在的点云数据的位置信息，确定所述第二点云数据与所述三维栅格空间中已存在的点云数据的空间分布相似度。According to the position information of the second point cloud data and the position information of the point cloud data that already exists in the three-dimensional grid space, determine the second point cloud data and the point cloud that already exists in the three-dimensional grid space The spatial distribution similarity of the data.
  37. 根据权利要求34所述的设备，其特征在于，所述处理器将所述第二点云数据投影至相机坐标系下的三维栅格空间之前，还用于：The device according to claim 34, wherein the processor is further configured to: before projecting the second point cloud data to a three-dimensional grid space in a camera coordinate system:
    确定所述相机的视角是否小于所述激光扫描装置的视角；Determining whether the angle of view of the camera is smaller than the angle of view of the laser scanning device;
    当确定所述相机的视角小于所述激光扫描装置的视角时，执行所述将所述第二点云数据投影至相机坐标系下的三维栅格空间的步骤。When it is determined that the angle of view of the camera is smaller than the angle of view of the laser scanning device, the step of projecting the second point cloud data to the three-dimensional grid space in the camera coordinate system is performed.
  38. 根据权利要求25所述的设备，其特征在于，所述满足预设条件，包括：The device according to claim 25, wherein said satisfying a preset condition comprises:
    所述投影三维空间中的每个栅格区域中的第二点云数据的数量大于预设数量阈值。The quantity of the second point cloud data in each grid area in the projection three-dimensional space is greater than a preset quantity threshold.
  39. 根据权利要求25所述的设备，其特征在于，所述处理器将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：The device according to claim 25, wherein the processor projects the projected three-dimensional space onto the image data collected by the camera, and obtains the most of the projected three-dimensional space onto the image data. In the optimal position, specifically used for:
    根据所述相机采集的图像数据，确定与所述图像数据对应的梯度图像；Determining a gradient image corresponding to the image data according to the image data collected by the camera;
    将所述投影三维空间的第二点云数据投影至所述梯度图像；Projecting the second point cloud data of the projected three-dimensional space onto the gradient image;
    当确定所述投影三维空间的第二点云数据投影至所述梯度图像，所述投影三维空间的第二点云数据与所述梯度图像完全融合时，确定所述投影三维空间投影到所述图像数据上的最优位置。When it is determined that the second point cloud data of the projected three-dimensional space is projected to the gradient image, and the second point cloud data of the projected three-dimensional space is completely fused with the gradient image, it is determined that the projected three-dimensional space is projected to the gradient image. The optimal position on the image data.
  40. 根据权利要求39所述的设备，其特征在于，所述处理器根据所述相机采集的图像数据，确定与所述图像数据对应的梯度图像时，具体用于：The device according to claim 39, wherein when the processor determines the gradient image corresponding to the image data according to the image data collected by the camera, it is specifically configured to:
    根据所述相机采集的图像数据，确定与所述图像数据对应的灰度图像；Determining a grayscale image corresponding to the image data according to the image data collected by the camera;
    从所述灰度图像中提取梯度信息和/或边缘信息；Extracting gradient information and/or edge information from the grayscale image;
    根据所述梯度信息和/或边缘信息，确定所述梯度图像。The gradient image is determined according to the gradient information and/or edge information.
  41. 根据权利要求25所述的设备，其特征在于，所述处理器将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：The device according to claim 25, wherein the processor projects the projected three-dimensional space onto the image data collected by the camera, and obtains the most of the projected three-dimensional space onto the image data. In the optimal position, it is specifically used for:
    获取将所述投影三维空间投影至所述相机采集的图像数据上得到的目标图像；Acquiring a target image obtained by projecting the projected three-dimensional space onto the image data collected by the camera;
    确定所述目标图像中第二点云数据的反射率；Determining the reflectivity of the second point cloud data in the target image;
    确定与所述目标图像对应的灰度图像的灰度值；Determining the gray value of the gray image corresponding to the target image;
    根据所述目标图像中第二点云数据的反射率以及与所述目标图像对应的灰度图像的灰度值，确定所述投影三维空间投影到所述图像数据上的最优位 置。According to the reflectivity of the second point cloud data in the target image and the gray value of the gray image corresponding to the target image, the optimal position of the projection three-dimensional space projected onto the image data is determined.
  42. 根据权利要求26所述的设备，其特征在于，所述处理器将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：The device according to claim 26, wherein the processor projects the projected three-dimensional space onto the image data collected by the camera, and obtains the most of the projected three-dimensional space onto the image data. In the optimal position, specifically used for:
    获取所述可移动平台在移动过程中的运动信息；Acquiring movement information of the movable platform during the movement;
    根据所述运动信息，确定所述第二点云数据的补偿信息；Determine the compensation information of the second point cloud data according to the motion information;
    根据所述补偿信息对所述投影三维空间中的第二点云数据进行补偿；Compensate the second point cloud data in the projected three-dimensional space according to the compensation information;
    将补偿后的所述第二点云数据投影至所述相机采集的图像数据上，以获取所述投影三维空间投影到所述图像数据上的最优位置。Projecting the compensated second point cloud data onto the image data collected by the camera to obtain the optimal position of the projected three-dimensional space projected onto the image data.
  43. 根据权利要求42所述的设备，其特征在于，The device of claim 42, wherein:
    所述运动信息包括位置信息、速度信息、加速度信息中的任意一种或多种。The motion information includes any one or more of position information, speed information, and acceleration information.
  44. 根据权利要求25所述的设备，其特征在于，所述处理器将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：The device according to claim 25, wherein the processor projects the projected three-dimensional space onto the image data collected by the camera, and obtains the most projected three-dimensional space onto the image data. In the optimal position, specifically used for:
    获取所述可移动平台在预设时间范围内移动过程中的第二点云数据；Acquiring second point cloud data during the movement of the movable platform within a preset time range;
    将在所述预设时间范围内获取到的所述投影三维空间中的第二点云数据投影至所述相机采集的图像数据上，以获取所述投影三维空间投影到所述图像数据上的最优位置。Projecting the second point cloud data in the projected three-dimensional space acquired within the preset time range onto the image data collected by the camera to obtain the projected three-dimensional space projected on the image data Optimal location.
  45. 根据权利要求42或44所述的设备，其特征在于，所述处理器将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：The device according to claim 42 or 44, wherein the processor projects the projected three-dimensional space onto the image data collected by the camera, and obtains the projected three-dimensional space projected onto the image data When the optimal position is specifically used for:
    获取所述可移动平台在移动过程中将所述投影三维空间投影至所述相机采集的图像数据上得到的多个目标图像；Acquiring a plurality of target images obtained by projecting the projected three-dimensional space onto the image data collected by the camera during the movement of the movable platform;
    将每个目标图像的数据进行比较；Compare the data of each target image;
    如果确定每个目标图像的数据一致，则确定所述目标图像的位置信息为所述投影三维空间投影到所述图像数据上的最优位置。If it is determined that the data of each target image is consistent, it is determined that the position information of the target image is the optimal position of the projection three-dimensional space projected onto the image data.
  46. 根据权利要求45所述的设备，其特征在于，所述处理器还用于：The device according to claim 45, wherein the processor is further configured to:
    如果确定每个所述目标图像的数据不一致，则确定激光扫描装置的外参发生变化，并对所述激光扫描装置的外参进行更新。If it is determined that the data of each target image is inconsistent, it is determined that the external parameters of the laser scanning device have changed, and the external parameters of the laser scanning device are updated.
  47. 根据权利要求45所述的设备，其特征在于，所述处理器还用于：The device according to claim 45, wherein the processor is further configured to:
    如果确定每个所述目标图像的数据不一致，则触发预设的报警装置进行报警，以提示用户对激光扫描装置进行检查。If it is determined that the data of each target image is inconsistent, a preset alarm device is triggered to give an alarm to prompt the user to check the laser scanning device.
  48. 根据权利要求25所述的设备，其特征在于，The device according to claim 25, wherein:
    所述激光扫描装置包括激光雷达、毫米波雷达、超声波雷达中的任意一种或多种。The laser scanning device includes any one or more of laser radar, millimeter wave radar, and ultrasonic radar.
  49. 一种可移动平台，其特征在于，包括：A movable platform, characterized in that it comprises:
    机身；body;
    配置在机身上的动力系统，用于为所述可移动平台提供移动的动力；The power system configured on the fuselage is used to provide mobile power for the movable platform;
    处理器，用于获取激光扫描装置采集的所述可移动平台周围环境的第一点云数据以及相机采集的图像数据；根据所述第一点云数据确定第二点云数据，所述第二点云数据用于指示无效的点云数据和/或不连续的点云数据；将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间；当所述投影三维空间中的每个栅格区域满足预设条件时，将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置。The processor is configured to obtain the first point cloud data of the surrounding environment of the movable platform collected by the laser scanning device and the image data collected by the camera; determine the second point cloud data according to the first point cloud data, the second The point cloud data is used to indicate invalid point cloud data and/or discontinuous point cloud data; the second point cloud data is projected to the three-dimensional raster space in the camera coordinate system to obtain the projected three-dimensional space; when the projection When each grid area in the three-dimensional space meets a preset condition, project the projected three-dimensional space onto the image data collected by the camera, and obtain the optimal position of the projected three-dimensional space onto the image data .
  50. 根据权利要求49所述的可移动平台，其特征在于，所述处理器根据所述第一点云数据确定第二点云数据时，具体用于：The mobile platform according to claim 49, wherein when the processor determines the second point cloud data according to the first point cloud data, it is specifically configured to:
    确定所述第一点云数据中相邻两个所述第一点云数据之间的距离；Determining the distance between two adjacent first point cloud data in the first point cloud data;
    根据所述相邻两个所述第一点云数据之间的距离，确定不连续的第二点云数据。According to the distance between the two adjacent first point cloud data, the discontinuous second point cloud data is determined.
  51. 根据权利要求50所述的可移动平台，其特征在于，所述处理器根据所述相邻两个所述第一点云数据之间的距离，确定不连续的第二点云数据时，具体用于：The mobile platform according to claim 50, wherein the processor determines the discontinuous second point cloud data according to the distance between the two adjacent first point cloud data, specifically Used for:
    确定所述相邻两个所述第一点云数据之间的距离是否大于第一预设阈值；Determining whether the distance between the two adjacent first point cloud data is greater than a first preset threshold;
    当确定出所述相邻两个所述第一点云数据之间的距离大于第一预设阈值时，确定所述相邻两个所述第一点云数据为不连续的第二点云数据。When it is determined that the distance between the two adjacent first point cloud data is greater than a first preset threshold, it is determined that the two adjacent first point cloud data are discontinuous second point clouds data.
  52. 根据权利要求50所述的可移动平台，其特征在于，所述处理器还用于：The movable platform of claim 50, wherein the processor is further configured to:
    获取所述第一点云数据和原点之间的距离；Acquiring the distance between the first point cloud data and the origin;
    根据所述第一点云数据和原点之间的距离以及所述相邻两个所述第一点云数据之间的距离，确定所述不连续的第二点云数据。Determine the discontinuous second point cloud data according to the distance between the first point cloud data and the origin and the distance between the two adjacent first point cloud data.
  53. 根据权利要求52所述的可移动平台，其特征在于，所述处理器根据所述第一点云数据和原点之间的距离以及所述相邻两个所述第一点云数据之间的距离，确定所述不连续的第二点云数据时，具体用于：The mobile platform of claim 52, wherein the processor is based on the distance between the first point cloud data and the origin and the distance between the two adjacent first point cloud data The distance, when determining the discontinuous second point cloud data, is specifically used for:
    确定与原点之间的距离大于预设值的第一点云数据；Determine the first point cloud data whose distance from the origin is greater than a preset value;
    从所述与原点之间的距离大于预设值的第一点云数据中，确定相邻两个所述第一点云数据之间的距离是否大于预设距离阈值；From the first point cloud data whose distance from the origin is greater than a preset value, determine whether the distance between two adjacent first point cloud data is greater than a preset distance threshold;
    若大于所述预设距离阈值，则确定所述相邻两个所述第一点云数据为不连续的第二点云数据。If it is greater than the preset distance threshold, it is determined that the two adjacent first point cloud data are discontinuous second point cloud data.
  54. 根据权利要求49所述的可移动平台，其特征在于，所述处理器根据所述第一点云数据确定第二点云数据时，具体用于：The mobile platform according to claim 49, wherein when the processor determines the second point cloud data according to the first point cloud data, it is specifically configured to:
    确定所述第一点云数据中是否存在深度信息；Determining whether depth information exists in the first point cloud data;
    根据所述深度信息确定所述第一点云数据中为无效的所述第二点云数据。The invalid second point cloud data in the first point cloud data is determined according to the depth information.
  55. 根据权利要求54所述的可移动平台，其特征在于，所述处理器根据所述深度信息确定所述第二点云数据时，具体用于：The mobile platform according to claim 54, wherein when the processor determines the second point cloud data according to the depth information, it is specifically configured to:
    从所述第一点云数据中确定不存在深度信息的所述第一点云数据为无效 的所述第二点云数据。It is determined from the first point cloud data that the first point cloud data without depth information is the invalid second point cloud data.
  56. 根据权利要求54所述的可移动平台，其特征在于，所述处理器根据所述深度信息确定所述第二点云数据时，具体用于：The mobile platform according to claim 54, wherein when the processor determines the second point cloud data according to the depth information, it is specifically configured to:
    获取所述第一点云数据的深度信息的变化值；Acquiring a change value of the depth information of the first point cloud data;
    当所述第一点云数据的深度信息的变化值大于第二预设阈值时，确定所述大于第二预设阈值对应的所述第一点云数据为无效的所述第二点云数据。When the change value of the depth information of the first point cloud data is greater than a second preset threshold, it is determined that the first point cloud data corresponding to the greater than the second preset threshold is invalid second point cloud data .
  57. 根据权利要求49所述的可移动平台，其特征在于，所述处理器根据所述第一点云数据确定第二点云数据之前，还用于：The mobile platform according to claim 49, wherein before determining the second point cloud data according to the first point cloud data, the processor is further configured to:
    将获取到的当前帧的第一点云数据与已经获取到的所述第一点云数据进行匹配；Matching the acquired first point cloud data of the current frame with the already acquired first point cloud data;
    确定所述当前帧的第一点云数据的空间分布与所述已经获取到的所述第一点云数据的空间分布的相似度；Determining the degree of similarity between the spatial distribution of the first point cloud data of the current frame and the spatial distribution of the acquired first point cloud data;
    如果所述相似度大于预设相似度阈值，则删除所述当前帧的第一点云数据；If the similarity is greater than the preset similarity threshold, delete the first point cloud data of the current frame;
    如果所述相似度小于或等于所述预设相似度阈值，则确定将所述当前帧的第一点云数据加入已经获取到的所述第一点云数据。If the similarity is less than or equal to the preset similarity threshold, it is determined to add the first point cloud data of the current frame to the first point cloud data that has been acquired.
  58. 根据权利要求49所述的可移动平台，其特征在于，所述处理器将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间时，具体用于：The movable platform according to claim 49, wherein the processor projects the second point cloud data to a three-dimensional grid space in a camera coordinate system, and when obtaining a projected three-dimensional space, it is specifically used for:
    确定所述激光扫描装置和相机之间的相对位置信息；Determining the relative position information between the laser scanning device and the camera;
    根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间。According to the relative position information, the second point cloud data is projected to the three-dimensional grid space in the camera coordinate system to obtain the projected three-dimensional space.
  59. 根据权利要求58所述的可移动平台，其特征在于，所述处理器根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间之前，还用于：The movable platform according to claim 58, wherein the processor projects the second point cloud data to a three-dimensional grid space in a camera coordinate system according to the relative position information, before obtaining the projected three-dimensional space , Also used for:
    确定所述第二点云数据与所述三维栅格空间中已存在的点云数据的空间 分布相似度；Determining the spatial distribution similarity between the second point cloud data and the point cloud data that already exists in the three-dimensional grid space;
    删除所述空间分布相似度大于预设相似度阈值的第二点云数据；Deleting the second point cloud data whose spatial distribution similarity is greater than a preset similarity threshold;
    所述处理器根据所述相对位置信息将所述第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间时，具体用于：When the processor projects the second point cloud data to the three-dimensional grid space in the camera coordinate system according to the relative position information, when obtaining the projected three-dimensional space, it is specifically used for:
    根据所述相对位置信息将所述删除后的第二点云数据投影至相机坐标系下的三维栅格空间，得到投影三维空间。Projecting the deleted second point cloud data to the three-dimensional grid space in the camera coordinate system according to the relative position information to obtain the projected three-dimensional space.
  60. 根据权利要求59所述的可移动平台，其特征在于，所述处理器确定所述第二点云数据与所述三维栅格空间中已存在的点云数据的空间分布相似度时，具体用于：The mobile platform according to claim 59, wherein when the processor determines the spatial distribution similarity between the second point cloud data and the point cloud data that already exists in the three-dimensional grid space, it specifically uses in:
    确定所述第二点云数据的位置信息以及所述三维栅格空间中已存在的点云数据的位置信息；Determining the position information of the second point cloud data and the position information of the point cloud data that already exists in the three-dimensional grid space;
    根据所述第二点云数据的位置信息与所述三维栅格空间中已存在的点云数据的位置信息，确定所述第二点云数据与所述三维栅格空间中已存在的点云数据的空间分布相似度。According to the position information of the second point cloud data and the position information of the point cloud data that already exists in the three-dimensional grid space, determine the second point cloud data and the point cloud that already exists in the three-dimensional grid space The spatial distribution similarity of the data.
  61. 根据权利要求60所述的可移动平台，其特征在于，所述处理器将所述第二点云数据投影至相机坐标系下的三维栅格空间之前，还用于：The mobile platform according to claim 60, wherein before the processor projects the second point cloud data to a three-dimensional grid space in a camera coordinate system, it is further used for:
    确定所述相机的视角是否小于所述激光扫描装置的视角；Determining whether the angle of view of the camera is smaller than the angle of view of the laser scanning device;
    当确定所述相机的视角小于所述激光扫描装置的视角时，执行所述将所述第二点云数据投影至相机坐标系下的三维栅格空间的步骤。When it is determined that the angle of view of the camera is smaller than the angle of view of the laser scanning device, the step of projecting the second point cloud data to the three-dimensional grid space in the camera coordinate system is performed.
  62. 根据权利要求49所述的可移动平台，其特征在于，所述满足预设条件，包括：The movable platform according to claim 49, wherein said meeting preset conditions comprises:
    所述投影三维空间中的每个栅格区域中的第二点云数据的数量大于预设数量阈值。The quantity of the second point cloud data in each grid area in the projection three-dimensional space is greater than a preset quantity threshold.
  63. 根据权利要求49所述的可移动平台，其特征在于，所述处理器将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：The movable platform according to claim 49, wherein the processor projects the projected three-dimensional space onto the image data collected by the camera, and obtains the projected three-dimensional space projected onto the image data When the optimal position is specifically used for:
    根据所述相机采集的图像数据，确定与所述图像数据对应的梯度图像；Determining a gradient image corresponding to the image data according to the image data collected by the camera;
    将所述投影三维空间的第二点云数据投影至所述梯度图像；Projecting the second point cloud data of the projected three-dimensional space onto the gradient image;
    当确定所述投影三维空间的第二点云数据投影至所述梯度图像，所述投影三维空间的第二点云数据与所述梯度图像完全融合时，确定所述投影三维空间投影到所述图像数据上的最优位置。When it is determined that the second point cloud data of the projected three-dimensional space is projected to the gradient image, and the second point cloud data of the projected three-dimensional space is completely fused with the gradient image, it is determined that the projected three-dimensional space is projected to the gradient image. The optimal position on the image data.
  64. 根据权利要求63所述的可移动平台，其特征在于，所述处理器根据所述相机采集的图像数据，确定与所述图像数据对应的梯度图像时，具体用于：The movable platform according to claim 63, wherein the processor is specifically configured to: when determining the gradient image corresponding to the image data according to the image data collected by the camera:
    根据所述相机采集的图像数据，确定与所述图像数据对应的灰度图像；Determining a grayscale image corresponding to the image data according to the image data collected by the camera;
    从所述灰度图像中提取梯度信息和/或边缘信息；Extracting gradient information and/or edge information from the grayscale image;
    根据所述梯度信息和/或边缘信息，确定所述梯度图像。The gradient image is determined according to the gradient information and/or edge information.
  65. 根据权利要求49所述的可移动平台，其特征在于，所述处理器将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：The movable platform according to claim 49, wherein the processor projects the projected three-dimensional space onto the image data collected by the camera, and obtains the projected three-dimensional space projected onto the image data When the optimal position is specifically used for:
    获取将所述投影三维空间投影至所述相机采集的图像数据上得到的目标图像；Acquiring a target image obtained by projecting the projected three-dimensional space onto the image data collected by the camera;
    确定所述目标图像中第二点云数据的反射率；Determining the reflectivity of the second point cloud data in the target image;
    确定与所述目标图像对应的灰度图像的灰度值；Determining the gray value of the gray image corresponding to the target image;
    根据所述目标图像中第二点云数据的反射率以及与所述目标图像对应的灰度图像的灰度值，确定所述投影三维空间投影到所述图像数据上的最优位置。According to the reflectance of the second point cloud data in the target image and the gray value of the gray image corresponding to the target image, the optimal position of the projection three-dimensional space projected onto the image data is determined.
  66. 根据权利要求50所述的可移动平台，其特征在于，所述处理器将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：The movable platform according to claim 50, wherein the processor projects the projected three-dimensional space onto the image data collected by the camera, and obtains the projected three-dimensional space projected onto the image data When the optimal position is specifically used for:
    获取所述可移动平台在移动过程中的运动信息；Acquiring movement information of the movable platform during the movement;
    根据所述运动信息，确定所述第二点云数据的补偿信息；Determine the compensation information of the second point cloud data according to the motion information;
    根据所述补偿信息对所述投影三维空间中的第二点云数据进行补偿；Compensate the second point cloud data in the projected three-dimensional space according to the compensation information;
    将补偿后的所述第二点云数据投影至所述相机采集的图像数据上，以获取 所述投影三维空间投影到所述图像数据上的最优位置。Projecting the compensated second point cloud data onto the image data collected by the camera to obtain an optimal position of the projected three-dimensional space projected onto the image data.
  67. 根据权利要求66所述的可移动平台，其特征在于，The movable platform according to claim 66, wherein:
    所述运动信息包括位置信息、速度信息、加速度信息中的任意一种或多种。The motion information includes any one or more of position information, speed information, and acceleration information.
  68. 根据权利要求49所述的可移动平台，其特征在于，所述处理器将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：The movable platform according to claim 49, wherein the processor projects the projected three-dimensional space onto the image data collected by the camera, and obtains the projected three-dimensional space projected onto the image data When the optimal position is specifically used for:
    获取所述可移动平台在预设时间范围内移动过程中的第二点云数据；Acquiring second point cloud data during the movement of the movable platform within a preset time range;
    将在所述预设时间范围内获取到的所述投影三维空间中的第二点云数据投影至所述相机采集的图像数据上，以获取所述投影三维空间投影到所述图像数据上的最优位置。Projecting the second point cloud data in the projected three-dimensional space acquired within the preset time range onto the image data collected by the camera to obtain the projected three-dimensional space projected on the image data Optimal location.
  69. 根据权利要求66或68所述的可移动平台，其特征在于，所述处理器将所述投影三维空间投影至所述相机采集的图像数据上，并获取所述投影三维空间投影到所述图像数据上的最优位置时，具体用于：The movable platform according to claim 66 or 68, wherein the processor projects the projected three-dimensional space onto the image data collected by the camera, and obtains the projected three-dimensional space projected onto the image When the optimal position on the data, specifically used for:
    获取所述可移动平台在移动过程中将所述投影三维空间投影至所述相机采集的图像数据上得到的多个目标图像；Acquiring a plurality of target images obtained by projecting the projected three-dimensional space onto the image data collected by the camera during the movement of the movable platform;
    将每个目标图像的数据进行比较；Compare the data of each target image;
    如果确定每个目标图像的数据一致，则确定所述目标图像的位置信息为所述投影三维空间投影到所述图像数据上的最优位置。If it is determined that the data of each target image is consistent, it is determined that the position information of the target image is the optimal position of the projection three-dimensional space projected onto the image data.
  70. 根据权利要求69所述的可移动平台，其特征在于，所述处理器还用于：The movable platform according to claim 69, wherein the processor is further configured to:
    如果确定每个所述目标图像的数据不一致，则确定激光扫描装置的外参发生变化，并对所述激光扫描装置的外参进行更新。If it is determined that the data of each target image is inconsistent, it is determined that the external parameters of the laser scanning device have changed, and the external parameters of the laser scanning device are updated.
  71. 根据权利要求69所述的可移动平台，其特征在于，所述处理器还用于：The movable platform according to claim 69, wherein the processor is further configured to:
    如果确定每个所述目标图像的数据不一致，则触发预设的报警装置进行报 警，以提示用户对激光扫描装置进行检查。If it is determined that the data of each target image is inconsistent, the preset alarm device is triggered to give an alarm to prompt the user to check the laser scanning device.
  72. 根据权利要求49所述的可移动平台，其特征在于，The movable platform of claim 49, wherein:
    所述激光扫描装置包括激光雷达、毫米波雷达、超声波雷达中的任意一种或多种。The laser scanning device includes any one or more of laser radar, millimeter wave radar, and ultrasonic radar.
  73. 根据权利要求49所述的可移动平台，其特征在于，The movable platform of claim 49, wherein:
    所述激光扫描装置和所述相机分别与所述可移动平台可拆卸连接。The laser scanning device and the camera are respectively detachably connected with the movable platform.
  74. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现如权利要求1至24任一项所述方法。A computer-readable storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 24 when the computer program is executed by a processor.
PCT/CN2019/098354 2019-07-30 2019-07-30 Calibration method and device, movable platform, and storage medium WO2021016854A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201980030471.8A CN112106111A (en) 2019-07-30 2019-07-30 Calibration method, calibration equipment, movable platform and storage medium
PCT/CN2019/098354 WO2021016854A1 (en) 2019-07-30 2019-07-30 Calibration method and device, movable platform, and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/098354 WO2021016854A1 (en) 2019-07-30 2019-07-30 Calibration method and device, movable platform, and storage medium

Publications (1)

Publication Number Publication Date
WO2021016854A1 true WO2021016854A1 (en) 2021-02-04

Family

ID=73748811

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/098354 WO2021016854A1 (en) 2019-07-30 2019-07-30 Calibration method and device, movable platform, and storage medium

Country Status (2)

Country Link
CN (1) CN112106111A (en)
WO (1) WO2021016854A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
ES2858148A1 (en) * 2021-05-27 2021-09-29 Univ Madrid Politecnica ADJUSTABLE MULTIPANEL EQUIPMENT FOR MOVEMENT CALIBRATION FROM POINT CLOUDS OBTAINED WITH "LASER LAND SCANNER (TLS)" IN THE FIELD (Machine-translation by Google Translate, not legally binding)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112578356A (en) * 2020-12-25 2021-03-30 上海商汤临港智能科技有限公司 External parameter calibration method and device, computer equipment and storage medium
CN114756162B (en) * 2021-01-05 2023-09-05 成都极米科技股份有限公司 Touch system and method, electronic device and computer readable storage medium
CN113639685B (en) * 2021-08-10 2023-10-03 杭州申昊科技股份有限公司 Displacement detection method, device, equipment and storage medium
CN113740829A (en) * 2021-11-05 2021-12-03 新石器慧通（北京）科技有限公司 External parameter monitoring method and device for environment sensing equipment, medium and running device

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150310135A1 (en) * 2014-04-24 2015-10-29 The Board Of Trustees Of The University Of Illinois 4d vizualization of building design and construction modeling with photographs
CN107564069A (en) * 2017-09-04 2018-01-09 北京京东尚科信息技术有限公司 The determination method, apparatus and computer-readable recording medium of calibrating parameters
CN108406731A (en) * 2018-06-06 2018-08-17 珠海市微半导体有限公司 A kind of positioning device, method and robot based on deep vision
CN109300162A (en) * 2018-08-17 2019-02-01 浙江工业大学 A kind of multi-line laser radar and camera combined calibrating method based on fining radar scanning marginal point
WO2019035049A1 (en) * 2017-08-16 2019-02-21 Mako Surgical Corp. Ultrasound bone registration with learning-based segmentation and sound speed calibration

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150310135A1 (en) * 2014-04-24 2015-10-29 The Board Of Trustees Of The University Of Illinois 4d vizualization of building design and construction modeling with photographs
WO2019035049A1 (en) * 2017-08-16 2019-02-21 Mako Surgical Corp. Ultrasound bone registration with learning-based segmentation and sound speed calibration
CN107564069A (en) * 2017-09-04 2018-01-09 北京京东尚科信息技术有限公司 The determination method, apparatus and computer-readable recording medium of calibrating parameters
CN108406731A (en) * 2018-06-06 2018-08-17 珠海市微半导体有限公司 A kind of positioning device, method and robot based on deep vision
CN109300162A (en) * 2018-08-17 2019-02-01 浙江工业大学 A kind of multi-line laser radar and camera combined calibrating method based on fining radar scanning marginal point

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
ES2858148A1 (en) * 2021-05-27 2021-09-29 Univ Madrid Politecnica ADJUSTABLE MULTIPANEL EQUIPMENT FOR MOVEMENT CALIBRATION FROM POINT CLOUDS OBTAINED WITH "LASER LAND SCANNER (TLS)" IN THE FIELD (Machine-translation by Google Translate, not legally binding)

Also Published As

Publication number Publication date
CN112106111A (en) 2020-12-18

Similar Documents

Publication Publication Date Title
WO2021016854A1 (en) Calibration method and device, movable platform, and storage medium
CN112017251B (en) Calibration method and device, road side equipment and computer readable storage medium
WO2020215172A1 (en) Obstacle detection method and device, mobile platform, and storage medium
KR102159376B1 (en) Laser scanning system, laser scanning method, mobile laser scanning system and program
TWI624170B (en) Image scanning system and method thereof
Kang et al. Automatic targetless camera–lidar calibration by aligning edge with gaussian mixture model
WO2021189468A1 (en) Attitude correction method, apparatus and system for laser radar
WO2021195939A1 (en) Calibrating method for external parameters of binocular photographing device, movable platform and system
CN113111513B (en) Sensor configuration scheme determining method and device, computer equipment and storage medium
CN114581480B (en) Multi-unmanned aerial vehicle cooperative target state estimation control method and application thereof
CN112146848A (en) Method and device for determining distortion parameter of camera
WO2020258217A1 (en) State estimation method and system for movable platform, movable platform, and storage medium
CN111142514B (en) Robot and obstacle avoidance method and device thereof
CN115359130A (en) Radar and camera combined calibration method and device, electronic equipment and storage medium
WO2022179207A1 (en) Window occlusion detection method and apparatus
CN111553342B (en) Visual positioning method, visual positioning device, computer equipment and storage medium
KR20200076628A (en) Location measuring method of mobile device, location measuring device and electronic device
CN114782556B (en) Camera and laser radar registration method and system and storage medium
CN113014899B (en) Binocular image parallax determination method, device and system
US20210149412A1 (en) Position estimating apparatus, method for determining position of movable apparatus, and non-transitory computer readable medium
CN112598736A (en) Map construction based visual positioning method and device
Weinmann et al. A step towards dynamic scene analysis with active multi-view range imaging systems
JP2021012043A (en) Information processing device for machine learning, information processing method for machine learning, and information processing program for machine learning
US20230419468A1 (en) Image processing apparatus, image processing method, and image processing program
US20220291009A1 (en) Information processing apparatus, information processing method, and storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19939805

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19939805

Country of ref document: EP

Kind code of ref document: A1