WO2022256976A1

WO2022256976A1 - Method and system for constructing dense point cloud truth value data and electronic device

Info

Publication number: WO2022256976A1
Application number: PCT/CN2021/098657
Authority: WO
Inventors: 宫正
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2021-06-07
Filing date: 2021-06-07
Publication date: 2022-12-15

Abstract

A method and system for constructing dense point cloud truth value data and an electronic device. The method comprises: obtaining image data acquired by a camera, point cloud data acquired by at least two distance measurement apparatuses and pose data acquired by an inertial measurement apparatus, the described sensors being disposed on the same movable platform, and the field of view of the camera at least partially coinciding with the fields of view of the at least two distance measurement apparatuses; performing time alignment and space alignment on the image data, the point cloud data and the pose data; extracting target point cloud data within a preset time range, and performing distortion correction on a motion distortion in the target point cloud data according to a change in the pose data to obtain corrected target point cloud data; performing noise filtering processing on the corrected target point cloud data to obtain noise-filtered target point cloud data; and outputting dense point cloud truth value data, the dense point cloud truth value data comprising the noise-filtered target point cloud data and space-aligned image data. By means of the method, dense and high-accuracy point cloud truth value data can be constructed.

Description

Method, system and electronic device for constructing dense point cloud true value data

manual

technical field

Embodiments of the present invention relate to the technical field of distance measurement, and more specifically, relate to a method, system and electronic device for constructing dense point cloud truth data.

Background technique

Three-dimensional point cloud detection systems such as lidar, laser rangefinders and other distance measuring devices can measure the time of light propagation between the distance measuring device and the measured object, that is, the time of flight (Time-of-Flight, TOF) , to detect the distance from the measured object to the distance measuring device.

The point cloud data collected by the ranging device is sparse and unevenly distributed. How to combine image information to construct a complete, high-precision and dense point cloud data is an urgent problem to be solved. Deep learning is considered to be an effective solution, which trains a specific network through a large amount of data, so as to obtain a network model that can generate dense point cloud data based on dense images and sparse point cloud data. Deep learning requires a large amount of data for training to improve the estimation accuracy of the network. Therefore, how to automatically construct dense point cloud true value data is one of the most important problems to be solved at present.

Contents of the invention

A series of concepts in simplified form are introduced in the Summary of the Invention, which will be further detailed in the Detailed Description. The summary of the invention in the present invention does not mean to limit the key features and essential technical features of the claimed technical solution, nor does it mean to try to determine the protection scope of the claimed technical solution.

In view of the deficiencies of the prior art, the first aspect of the embodiment of the present invention provides a method for constructing dense point cloud truth data, including:

Obtain image data collected by the camera, point cloud data collected by at least two distance measuring devices, and pose data of the camera and the distance measuring device collected by the inertial measurement device, the camera, the at least two distance measuring devices The device and the inertial measurement device are arranged on the same movable platform, and the fields of view of the camera and the at least two distance measuring devices at least partially overlap;

Acquiring time information of the image data, the point cloud data, and the pose data, and time-aligning the image data, the point cloud data, and the pose data according to the time information;

Calibrate the camera, the distance measuring device and the inertial measurement device to obtain the spatial transformation relationship between the image, the point cloud data and the pose data, and according to the spatial transformation relationship spatially aligning the image data, the point cloud data, and the pose data;

Extract the target point cloud data within the preset time range from the point cloud data transformed into the same space coordinate system, and perform distortion correction on the motion distortion in the target point cloud data according to the change of the pose data, so as to obtain The corrected target point cloud data;

Noise filtering is performed on the corrected target point cloud data to obtain the target point cloud data after noise filtering;

Outputting dense point cloud true value data, the dense point cloud true value data includes the target point cloud data after the noise filtering process and the image data after the spatial alignment.

The second aspect of the embodiment of the present invention provides a system for constructing dense point cloud true value data. The system includes a camera, at least two distance measuring devices, an inertial measurement device and a processor. The camera, the at least two The distance measuring device and the inertial measurement device are arranged on the same movable platform, and the fields of view of the camera and the at least two distance measuring devices at least partially overlap;

The processor is used to:

Acquiring the image data collected by the camera, the point cloud data collected by the at least two distance measuring devices, and the pose data of the camera and the distance measuring device collected by the inertial measurement device, the camera and the distance measuring device The fields of view of the at least two ranging devices at least partially overlap;

Output dense point cloud true value data, described dense point cloud true value data comprises the target point cloud data after the noise filtering process and the image data after carrying out the space alignment.

The third aspect of the embodiment of the present invention provides an electronic device, and the electronic device includes:

memory for storing executable instructions;

A processor, configured to execute the instructions stored in the memory, so that the processor executes the method for constructing dense point cloud truth data as described above

The method, system, and electronic device for constructing dense point cloud true value data in the embodiments of the present invention use point cloud data collected by multiple ranging devices to construct dense point clouds, which can avoid the problems of inconsistency in depth and blurring of moving objects caused by inconsistent viewing angles; through Time alignment, spatial alignment and noise filtering are performed on the point cloud data collected by multiple ranging devices, which improves the accuracy of the dense point cloud true value data.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without paying creative labor.

Fig. 1 is a schematic frame diagram of a distance measuring device involved in an embodiment of the present invention;

Fig. 2 is a schematic diagram of an embodiment in which the distance measuring device involved in the embodiment of the present invention adopts a coaxial optical path;

Fig. 3 is a schematic diagram of a scanning pattern of a ranging device according to an embodiment of the present invention;

Fig. 4 is a schematic flowchart of a method for constructing dense point cloud true value data according to an embodiment of the present invention;

5 is a schematic diagram of a construction system of dense point cloud true value data according to an embodiment of the present invention;

Fig. 6A is the target point cloud data before distortion correction is performed on the motion distortion caused by the motion of the distance measuring device according to an embodiment of the present invention;

Fig. 6B is the target point cloud data after distortion correction is performed on the motion distortion caused by the motion of the distance measuring device according to an embodiment of the present invention;

Fig. 7A is a point cloud cluster before distortion correction is performed on the motion distortion caused by the motion of the target object according to an embodiment of the present invention;

Fig. 7B is a point cloud cluster after distortion correction is performed on the motion distortion caused by the motion of the target object according to an embodiment of the present invention;

Fig. 8 is a schematic diagram of dense point cloud true value data according to an embodiment of the present invention;

Fig. 9 is a schematic block diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

In order to make the objects, technical solutions and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Apparently, the described embodiments are only some embodiments of the present invention, rather than all embodiments of the present invention, and it should be understood that the present invention is not limited by the exemplary embodiments described here. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative effort shall fall within the protection scope of the present invention.

In the following description, numerous specific details are given in order to provide a more thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without one or more of these details. In other examples, some technical features known in the art are not described in order to avoid confusion with the present invention.

It should be understood that the invention can be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the/the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the terms "consists of" and/or "comprising", when used in this specification, identify the presence of stated features, integers, steps, operations, elements and/or parts, but do not exclude one or more other Presence or addition of features, integers, steps, operations, elements, parts and/or groups. As used herein, the term "and/or" includes any and all combinations of the associated listed items.

In order to thoroughly understand the present invention, a detailed structure will be provided in the following description to illustrate the technical solution proposed by the present invention. Alternative embodiments of the invention are described in detail below, however the invention may have other embodiments beyond these detailed descriptions.

The laser ranging method provided by various embodiments of the present invention can be applied to a ranging device, and the ranging device can be an electronic device such as a laser radar or a laser ranging device. In one embodiment, the ranging device is used to sense external environment information, for example, distance information, orientation information, reflection intensity information, speed information, etc. of environmental objects. The distance measuring device can detect the distance from the detection object to the distance measurement device by measuring the time of light propagation between the distance measurement device and the detection object, that is, the time-of-flight (TOF).

For ease of understanding, the working process of distance measurement will be described below with reference to the distance measurement device 100 shown in FIG. 1 .

As shown in FIG. 1 , the ranging device 100 may include a transmitting circuit 110 , a receiving circuit 120 , a sampling circuit 130 and an arithmetic circuit 140 .

The transmitting circuit 110 may transmit a sequence of light pulses (eg, a sequence of laser pulses). The receiving circuit 120 can receive the light pulse sequence reflected by the measured object, and perform photoelectric conversion on the light pulse sequence to obtain an electrical signal, and then process the electrical signal and output it to the sampling circuit 130 . The sampling circuit 130 can sample the electrical signal to obtain a sampling result. The arithmetic circuit 140 can determine the distance between the distance measuring device 100 and the measured object based on the sampling result of the sampling circuit 130 .

Optionally, the distance measuring device 100 may further include a control circuit 150, which can control other circuits, for example, control the working time of each circuit and/or set parameters for each circuit.

It should be understood that although the ranging device shown in FIG. 1 includes a transmitting circuit, a receiving circuit, a sampling circuit and an arithmetic circuit for emitting a light beam for detection, the embodiment of the present invention is not limited thereto. The transmitting circuit The number of any one of the receiving circuit, the sampling circuit, and the computing circuit can also be at least two, for emitting at least two light beams along the same direction or respectively along different directions; wherein, the at least two light paths can be simultaneously It can also be emitted at different times. In an example, the light emitting chips in the at least two emitting circuits are packaged in the same module. For example, each emitting circuit includes a laser emitting chip, and the dies of the laser emitting chips in the at least two emitting circuits are packaged together and accommodated in the same packaging space.

In some implementations, in addition to the circuit shown in FIG. 1 , the distance measuring device 100 may further include a scanning module 160 configured to change the propagation direction of at least one laser pulse sequence emitted by the transmitting circuit to emit.

Wherein, the module including the transmitting circuit 110, the receiving circuit 120, the sampling circuit 130 and the operation circuit 140, or the module including the transmitting circuit 110, the receiving circuit 120, the sampling circuit 130, the operation circuit 140 and the control circuit 150 may be called a measuring circuit. The ranging module 150 may be independent from other modules, for example, the scanning module 160 .

A coaxial optical path can be used in the distance measuring device, that is, the light beam emitted by the distance measuring device and the reflected light beam share at least part of the optical path in the distance measuring device. For example, after the at least one laser pulse sequence emitted by the transmitting circuit changes its propagation direction and exits through the scanning module, the laser pulse sequence reflected by the object under test passes through the scanning module and enters the receiving circuit. Alternatively, the distance measuring device may also adopt an off-axis optical path, that is, the light beam emitted by the distance measuring device and the reflected light beam are respectively transmitted along different optical paths in the distance measuring device. Fig. 2 shows a schematic diagram of an embodiment in which the distance measuring device of the present invention adopts a coaxial optical path.

The ranging device 200 includes a ranging module 210, and the ranging module 210 includes a transmitter 203 (which may include the above-mentioned transmitting circuit), a collimator element 204, a detector 205 (which may include the above-mentioned receiving circuit, sampling circuit and computing circuit) and The light path changing element 206 . The distance measuring module 210 is used for emitting light beams, receiving return light, and converting the return light into electrical signals. Wherein, the transmitter 203 can be used to transmit the light pulse sequence. In one embodiment, the transmitter 203 may emit a sequence of laser pulses. Optionally, the laser beam emitted by the transmitter 203 is a narrow-bandwidth beam whose wavelength is outside the range of visible light. The collimating element 204 is arranged on the outgoing light path of the emitter, and is used for collimating the light beam emitted from the emitter 203, and collimating the light beam emitted by the emitter 203 into a parallel light that is emitted to the scanning module. The collimating element is also used for converging at least a part of the returned light reflected by the measured object. The collimating element 204 may be a collimating lens or other elements capable of collimating light beams.

In the embodiment shown in FIG. 2, the transmitting optical path and receiving optical path in the distance measuring device are combined before the collimating element 204 through the optical path changing element 206, so that the transmitting optical path and the receiving optical path can share the same collimating element, so that the optical path more compact. In some other implementation manners, it is also possible that the emitter 203 and the detector 205 respectively use their own collimating elements, and the optical path changing element 206 is arranged on the optical path after the collimating element.

In the embodiment shown in Fig. 2, since the beam aperture of the light beam emitted by the transmitter 203 is relatively small, the beam aperture of the return light received by the distance measuring device is relatively large, so the optical path changing element can be realized by using a small-area reflector. Merge the transmit light path and the receive light path. In some other implementation manners, the optical path changing element may also use a reflector with a through hole, wherein the through hole is used to transmit the outgoing light of the emitter 203 , and the reflector is used to reflect the return light to the detector 205 . In this way, the shielding of the return light by the support of the small reflector in the case of using the small reflector can be reduced.

In the embodiment shown in FIG. 2 , the optical path changing element deviates from the optical axis of the collimating element 204 . In some other implementation manners, the optical path changing element may also be located on the optical axis of the collimating element 204 .

The ranging device 200 also includes a scanning module 202 . The scanning module 202 is placed on the outgoing optical path of the distance measuring module 210. The scanning module 202 is used to change the transmission direction of the collimated light beam 219 emitted by the collimating element 204 and project it to the external environment, and project the return light to the collimating element 204 . The returned light is converged onto the detector 205 through the collimation element 204 .

In one embodiment, the scanning module 202 may include at least one optical element for changing the propagation path of the beam, wherein the optical element may change the propagation path of the beam by reflecting, refracting, diffracting and so on. For example, the scanning module 202 includes lenses, mirrors, prisms, gratings, liquid crystals, optical phased arrays (Optical Phased Array), or any combination of the above optical elements. In an example, at least part of the optical elements are movable, for example, driven by a driving module to move the at least part of the optical elements, and the moving optical elements can reflect, refract or diffract light beams to different directions at different times. In some embodiments, multiple optical elements of scanning module 202 may rotate or vibrate about a common axis 209, with each rotating or vibrating optical element serving to continuously change the direction of propagation of the incident light beam. In one embodiment, the multiple optical elements of scanning module 202 may rotate at different rotational speeds, or vibrate at different speeds. In another embodiment, at least some of the optical elements of scanning module 202 may rotate at substantially the same rotational speed. In some embodiments, the multiple optical elements of the scanning module may also rotate about different axes. In some embodiments, the multiple optical elements of the scanning module may also rotate in the same direction or in different directions; or vibrate in the same direction or in different directions, which is not limited here.

In one embodiment, the scanning module 202 includes a first optical element 214 and a driver 216 connected to the first optical element 214, the driver 216 is used to drive the first optical element 214 to rotate around the rotation axis 209, so that the first optical element 214 changes The direction of the collimated light beam 219 . The first optical element 214 projects the collimated light beam 219 in different directions. In one embodiment, the angle between the direction of the collimated light beam 219 changed by the first optical element and the rotation axis 209 changes as the first optical element 214 rotates. In one embodiment, first optical element 214 includes a pair of opposing non-parallel surfaces through which collimated light beam 219 passes. In one embodiment, the first optical element 214 comprises a prism having a thickness varying along at least one radial direction. In one embodiment, the first optical element 114 includes a wedge prism that refracts the collimated light beam 219 .

In one embodiment, the scanning module 202 further includes a second optical element 215 , the second optical element 215 rotates around the rotation axis 209 , and the rotation speed of the second optical element 215 is different from that of the first optical element 214 . The second optical element 215 is used to change the direction of the light beam projected by the first optical element 214 . In one embodiment, the second optical element 215 is connected with another driver 217, and the driver 217 drives the second optical element 215 to rotate. The first optical element 214 and the second optical element 215 can be driven by the same or different drivers, so that the rotation speed and/or the direction of rotation of the first optical element 214 and the second optical element 215 are different, thereby projecting a collimated light beam 219 to the external space In different directions, a larger spatial range can be scanned. In one embodiment, the controller 218 controls the

drivers

216 and 217 to drive the first optical element 214 and the second optical element 215 respectively. The rotational speeds of the first optical element 214 and the second optical element 215 can be determined according to the area and pattern expected to be scanned in practical applications.

Drivers

216 and 217 may include motors or other drivers.

In one embodiment, the second optical element 215 includes a pair of opposing non-parallel surfaces through which the light beam passes. In one embodiment, the second optical element 215 comprises a prism whose thickness varies along at least one radial direction. In one embodiment, the second optical element 215 includes a wedge prism.

In one embodiment, the scanning module 202 further includes a third optical element (not shown in the figure) and a driver for driving the movement of the third optical element. Optionally, the third optical element comprises a pair of opposite non-parallel surfaces through which the light beam passes. In one embodiment, the third optical element comprises a prism whose thickness varies along at least one radial direction. In one embodiment, the third optical element comprises a wedge prism. At least two of the first, second and third optical elements rotate at different rotational speeds and/or deflections.

The rotation of each optical element in the scanning module 202 can project light to different directions, such as

directions

211 and 213 , so as to scan the space around the distance measuring device 200 . As shown in FIG. 3 , FIG. 3 is a schematic diagram of a scanning pattern of the ranging device 200 . It can be understood that when the speed of the optical elements in the scanning module changes, the scanning pattern will also change accordingly.

When the light 211 projected by the scanning module 202 hits the detection object 201 , a part of the light is reflected by the detection object 201 to the distance measuring device 200 in a direction opposite to the projected light 211 . The return light 212 reflected by the detection object 201 enters the collimation element 204 after passing through the scanning module 202 .

The detector 205 is placed on the same side of the collimation element 204 as the emitter 203, and the detector 205 is used to convert at least part of the return light passing through the collimation element 204 into an electrical signal.

In one embodiment, each optical element is coated with an anti-reflection film. Optionally, the thickness of the antireflection film is equal to or close to the wavelength of the light beam emitted by the emitter 203, which can increase the intensity of the transmitted light beam.

In one embodiment, a filter layer is coated on the surface of a component located on the beam propagation path in the ranging device, or an optical filter is arranged on the beam propagation path, for at least transmitting the wavelength band of the beam emitted by the transmitter, Reflect other bands to reduce noise from ambient light to the receiver.

In some embodiments, the transmitter 203 may include a laser diode, and the laser diode emits nanosecond-level laser pulses. Further, the laser pulse receiving time can be determined, for example, the laser pulse receiving time can be determined by detecting the rising edge time and/or falling edge time of the electrical signal pulse. In this way, the distance measuring device 200 can calculate the TOF by using the pulse receiving time information and the pulse sending time information, so as to determine the distance from the detection object 201 to the distance measuring device 200 .

The distance and orientation detected by the ranging device 200 can be used for remote sensing, obstacle avoidance, surveying and mapping, modeling, navigation and so on. In one embodiment, the distance measuring device in the embodiment of the present invention can be applied to a movable platform, and the distance measuring device can be installed on the movable platform body of the movable platform. The movable platform with the distance measuring device can measure the external environment, for example, measure the distance between the movable platform and obstacles for purposes such as obstacle avoidance, and perform two-dimensional or three-dimensional mapping of the external environment. In some embodiments, the mobile platform includes at least one of an unmanned aerial vehicle, an automobile, a remote control vehicle, a robot, and a camera. When the ranging device is applied to the unmanned aerial vehicle, the movable platform body is the fuselage of the unmanned aerial vehicle. When the distance measuring device is applied to a car, the movable platform body is the body of the car. The car may be an automatic driving car or a semi-automatic driving car, which is not limited here. When the distance measuring device is applied to the remote control car, the movable platform body is the body of the remote control car. When the ranging device is applied to a robot, the movable platform body is a robot.

Because there are foreground and background in different viewing angles of a single ranging device, there is inconsistency in its depth. If the dense point cloud true value data is constructed based on a single ranging device and using a mobile viewing angle, it is impossible or difficult to solve the depth consistency under different viewing angles. question. In addition, the point cloud data collected from different angles of view has a large error in the measurement depth of the moving target, which will cause the surface of the moving target to be blurred and severely distorted.

To this end, the embodiment of the present invention proposes a method for constructing dense point cloud true value data based on multiple distance measuring devices. FIG. 4 shows the construction method 400 of dense point cloud true value data according to an embodiment of the present invention. Schematic flow chart. As shown in FIG. 4, the construction method 400 of dense point cloud true value data includes the following steps:

In step S410, the image data collected by the camera, the point cloud data collected by at least two distance measuring devices, and the pose data of the camera and the distance measuring device collected by the inertial measurement device are acquired, the camera, the at least The two distance measuring devices and the inertial measurement device are arranged on the same movable platform, and the fields of view of the camera and the at least two distance measuring devices at least partially overlap;

In step S420, acquire the time information of the image data, the point cloud data and the pose data, and time-align the image data, the point cloud data and the pose data according to the time information ;

In step S430, the camera, the distance measuring device and the inertial measurement device are calibrated to obtain the spatial transformation relationship between the image, the point cloud data and the pose data, and according to the The image data, the point cloud data and the pose data are spatially aligned according to the spatial transformation relationship;

In step S440, the target point cloud data within a preset time range is extracted from the point cloud data transformed into the same space coordinate system, and the motion distortion in the target point cloud data is distorted according to the change of the pose data Correction to obtain the corrected target point cloud data;

In step S450, perform noise filtering processing on the corrected target point cloud data to obtain target point cloud data after noise filtering processing;

In step S460, the dense point cloud true value data is output, and the dense point cloud true value data includes the target point cloud data after the noise filtering process and the image data after the spatial alignment.

The method 400 for constructing dense point cloud true value data in the embodiment of the present invention constructs dense point cloud data based on point cloud data collected by multiple distance measuring devices, which can avoid the problems of inconsistency in depth and blurring of moving objects caused by inconsistent viewing angles of a single distance measuring device .

FIG. 5 shows a system 500 for constructing dense point cloud ground truth data according to an embodiment of the present invention. As shown in FIG. 5 , the camera 510 and at least two distance measuring devices 520 are arranged on the same movable platform, and the fields of view of the camera 510 and the at least two distance measuring devices 520 at least partially overlap. Among them, mobile platforms include cars, remote control cars, robots, unmanned aerial vehicles, etc. In order to better simulate real application scenarios, the camera 510 is controlled to collect image data, at least two ranging devices 520 are controlled to collect point cloud data, and the inertial measurement device 530 is controlled to collect pose data while the movable platform is in motion. The point cloud data collected by the ranging device 510 is a collection of a large number of point cloud points, wherein each point cloud point contains at least three-dimensional coordinate information of an object; the image collected by the camera is a two-dimensional image (for example, a grayscale image or an RGB image), Key features such as color and texture of objects can be provided.

Further, each distance measuring device 520 and camera 510 have the same orientation, and their layout fits each other as much as possible to obtain as large an overlapping field of view as possible, so as to collect point cloud data and images in the same field of view to the greatest extent data. The ranging device 520 may be implemented as the ranging device described with reference to FIG. 1 and FIG. 2 , and the camera 510 may be implemented as a monocular camera, specifically, a monocular visible light camera. In some embodiments, in order to increase the density of point cloud data, there are no less than three ranging devices 520 for collecting point cloud data.

In addition, an inertial measurement device 530 is also mounted on the movable platform, and the inertial measurement device 530 may be a high-precision inertial navigation system capable of obtaining a high-precision position and attitude with 6 degrees of freedom. Exemplarily, the inertial measurement device 530 includes, but is not limited to, a gyroscope and an accelerometer. The gyroscope is used to form a navigation coordinate system, stabilize the measurement axis of the accelerometer in this coordinate system, and give the heading and attitude angle; To measure the acceleration of the carrier, the acceleration can be integrated to obtain the velocity, and the velocity can be integrated to obtain the distance. The inertial measurement device 530 may be an inertial measurement device equipped on the movable platform itself. Since the inertial measurement device 530, camera 510, distance measuring device 520 and other sensors are all mounted on the movable platform, the pose data collected by the inertial measurement device 530 can be understood as the pose data of the camera 510 and the distance measuring device 520, the inertial The pose data collected by the measuring device 530 can be used to correct point cloud distortion caused by the movement of the distance measuring device 520 itself.

In step S420, the time information of the image data, point cloud data and pose data is acquired, and the image data, point cloud data and pose data are time-aligned according to the time information. Temporal information includes timestamps of image data, point cloud data, and pose data. Time alignment is to find the image data, point cloud data and pose data at the same time, so that the distortion of the point cloud data can be corrected according to the pose data, and the matching image data and point cloud data can be obtained, so as to form a Dense point cloud ground truth data for training network models.

Exemplarily, image data, point cloud data and pose data are data after time synchronization based on the data synchronization system 540 . After the data collected by the camera 510 , the distance measuring device 520 and the inertial measurement device 530 are acquired by the data synchronization system 540 , the time stamps of the data are unified in different time domains. The data synchronization system 540 transmits the time-synchronized data to the processor 550, and the processor 550 time-aligns the image data, point cloud data, and pose data according to the time information of the image data, point cloud data, and pose data, specifically Specifically, the time stamp of the point cloud data can be obtained, and the image data and pose data corresponding to the time can be found according to the time stamp, where the pose data can include the pose data corresponding to each moment, and the image data can include the corresponding image data.

Exemplarily, GPS (Global Positioning System, Global Positioning System) can be used for time synchronization, and the IEEE 1588 clock synchronization protocol based on Ethernet can be used for time synchronization, or based on PPS (Pulse per Second, pulse signal per second) for time synchronization Wait. Of course, the above time synchronization method is only an example and not a limitation, and any suitable time synchronization method can be applied to the method 400 for constructing dense point cloud ground truth data according to the embodiment of the present invention.

Since the data output by each sensor is referenced to the coordinate system of the sensor, the coordinate systems corresponding to the data output by different sensors are different. Therefore, after performing the time alignment as described above, step S430 is performed to calibrate the camera, distance measuring device and inertial measurement device to obtain the spatial transformation relationship between the image, point cloud data and pose data, and according to the spatial Transformation relations spatially align image data, point cloud data, and pose data. Among them, the calibration between different sensors is to calibrate the external parameters of different sensors; the spatial alignment is to unify the data of multiple sensors into the same spatial coordinate system. The calibration can be performed in real time or in advance.

Specifically, calibrating the camera, the distance measuring device and the inertial measurement device includes: calibrating at least two distance measuring devices with each other, calibrating the distance measuring device and the camera, and calibrating the distance measuring device and the inertial measurement device . Through the above calibration algorithm, the multi-sensor data is unified under the spatial coordinate system, so as to facilitate subsequent distortion correction of point cloud data and obtain point cloud data with consistent depth.

Wherein, the methods for calibrating the distance measuring device, the camera and the inertial measurement device include but not limited to manual calibration, semi-automatic calibration and fully automatic calibration. Taking automatic calibration as an example, it is possible to perform feature extraction and feature matching on point cloud data and images with specific markers placed in front of the distance measuring device and camera in advance, and then calibrate the distance between the distance measuring device and the camera based on the feature matching results. external reference. Taking manual calibration as an example, a user interaction interface can be provided. The user interaction interface includes a parameter input area and a display area. The parameter input area is used for the user to input the external parameters of the sensor, and the display area is used to display the data collected by each sensor. The data displayed in the display area adjusts the external parameters of each sensor. For example, for the point cloud data collected by the distance measuring device, the point cloud data output by each distance measuring device can be converted to the position in space through the external parameters of the corresponding distance measuring device, so as to realize the position of the point cloud in space convert. By observing the display area, the user can determine whether the point cloud data after position transformation is in the same coordinate system. The external parameters of the distance measuring devices are adjusted so that the point cloud data detected by the various distance measuring devices are located in the same coordinate system after conversion.

In one embodiment, the distance measuring device includes a first distance measuring device and at least one second distance measuring device, wherein the space coordinate system of the first point cloud data collected by the first distance measuring device is used as the reference coordinate system, then in the When at least two distance measuring devices are calibrated, each second distance measuring device and the first distance measuring device can be calibrated respectively to obtain the second point cloud data collected by each second distance measuring device and the first distance measuring device. The spatial transformation relationship between the first point cloud data collected by the device, that is, when the coordinate system of the point cloud data is unified, the point cloud data collected by all distance measuring devices are converted to the first point cloud collected by the first distance measuring device In the spatial coordinate system of the data. When calibrating the camera and the distance measuring device, the camera and the first distance measuring device can be calibrated to obtain the spatial transformation relationship between the image and the first point cloud data, thereby converting the image to the first point cloud data. in the space coordinate system. When calibrating the inertial measurement device and the distance measuring device, the inertial measurement device and the first distance measuring device can be calibrated to obtain the spatial transformation relationship between the pose data and the first point cloud data, so that the pose data Convert to the space coordinate system of the first point cloud data. Wherein, the first distance measuring device may be any one of at least two distance measuring devices. In other embodiments, the coordinate system of the image data may be used as the reference coordinate system, the coordinate system of the inertial measurement data may be used as the reference coordinate system, or the coordinate system of the movable platform may be used as the reference coordinate system, as long as the multi-sensor The data can be unified to the same coordinate system.

After unifying the point cloud data collected by multiple ranging devices into the same space coordinate system, dense point cloud data can be obtained. However, since the distance measuring device is based on its own coordinate system when calculating point cloud point coordinates, and the distance measuring device of the present application is mounted on a movable platform, during the movement of the movable platform, the difference between the point cloud data collected at different times The reference coordinate system is different, causing the point cloud data to be distorted. In order to solve this problem, it is necessary to obtain the motion information of the ranging device during the acquisition process, and convert the coordinates of each point cloud point to the target time point for motion compensation through the motion compensation algorithm, so as to use the point cloud data after motion compensation to construct Dense point cloud ground truth data.

Therefore, after time and space alignment, step S440 is executed to extract the target point cloud data within the preset time range from the point cloud data transformed into the same space coordinate system, and to adjust the target point cloud data according to the change of pose data Distortion correction is performed on the motion distortion in the image to obtain the corrected target point cloud data. In an example, the time length of the preset time range may be longer than the acquisition time of a single frame of point cloud data, so as to obtain dense point cloud data. In other examples, since the embodiment of the present invention adopts multiple ranging devices to jointly collect point cloud data, dense point cloud data can be obtained without too long a time, so the length of the preset time range can also be equal to or less than a single Acquisition time of frame point cloud data, thereby reducing motion distortion. Exemplarily, the preset time range is greater than or equal to 100ms.

After extracting the target point cloud data within the preset time range, the pose data aligned in time and space is applied to the target point cloud data to correct the motion distortion caused by the movement of the ranging device itself, that is, the target point cloud data Perform motion compensation. The reason for motion distortion is that the point cloud points in the target point cloud data are not collected at the same time, but are collected continuously within the preset time range. During the collection process, the distance measuring device moves with the movable platform. Since the point cloud point measures the distance between the object and the distance measuring device, for the same object, the depth measured at different times is inconsistent, which causes the point cloud data to be distorted. Figure 6A and Figure 6B respectively show the target point cloud data before and after distortion correction. It can be seen from Fig. 6A and Fig. 6B that distortion correction effectively reduces the smear phenomenon of point cloud data.

Specifically, since the pose data and the target point cloud data are located in the same space coordinate system, the movement direction and speed of the target point cloud data within the preset time range can be obtained according to the change of the pose data, and according to the obtained movement direction The motion distortion caused by the motion of the ranging device in the target point cloud data can be corrected for distortion. Among them, in order to obtain accurate motion direction and motion speed, the pose data for a long time can be extracted to determine the motion direction and motion speed of point cloud data.

Specifically, according to the time information of the target point cloud data, corresponding correction coefficients can be assigned to the target point cloud data at different times, and the correction coefficients can also be called distortion coefficients. Small; the larger the distortion, the larger the correction coefficient, and the larger the correction range required. After the correction coefficient is determined, the target point cloud coefficients at different moments can be corrected according to the correction coefficients corresponding to the target point cloud data at different moments and the moving direction and moving speed of the target point cloud data.

When determining the correction coefficient, it is necessary to determine a reference time point within the preset time range, and assign correction coefficients to the point cloud points collected at different times according to the interval between the collection time of point cloud points and the reference time point. According to the assumption of short-term uniform speed, the correction coefficients of all point cloud points in the target point cloud data can be evenly divided by time, and normalized to [0-1], so as to assign point cloud points according to time t(t0-tn) Corresponding correction factor. The closer the distance to the reference time point, the smaller the distortion and the closer the correction coefficient is to 0; on the contrary, the farther the distance from the reference time point is, the greater the distortion is and the closer the correction coefficient is to 1. After distributing the distortion coefficients, the motion interpolation estimation at the corresponding moment is applied to obtain the final distortion correction result.

For example, assuming that the moving direction of the target point cloud data is △q and the moving distance is △d, then for the i-th point in the target point cloud data, its original coordinate point(i) before distortion correction is obtained, and Obtain its time stamp ti, and assign a correction coefficient s to it according to the time stamp ti of the i-th point, where s=1-(ti-t0)/(tn–t0); after that, the rotation of the i-th point can be obtained according to the correction coefficient Quantity interpolation slerp(s,△q) and translation interpolation s*△d, where slerp represents spherical linear interpolation; finally, the coordinates of the i-th point after correction are slerp(s,△q)*point(i)+ s*△d.

As mentioned above, the distortion correction for the motion distortion caused by the movement of the ranging device is to perform motion compensation on the target point cloud data within the preset time range, so that the point cloud points collected at different times are all compensated to the point where the target time point is located. Location. In one embodiment, the target time point may be the starting moment of the preset time range, then performing distortion correction on the motion distortion caused by the movement of the distance measuring device in the target point cloud data includes: for each point cloud data in the target According to the pose data, estimate its position at the initial moment of the preset time range, so as to correct the target point cloud data to the position corresponding to the initial moment of the preset time range. In other embodiments, the target point cloud data may also be corrected to a position corresponding to the end moment of the preset time range, or a position corresponding to any other moment in the preset time range.

Furthermore, since the target point cloud data is collected within a period of time, if there is a moving object in the field of view, the point cloud cluster of the moving object contained in the point cloud data will also have motion distortion. Therefore, after the distortion correction is performed on the distortion caused by the movement of the ranging device, the distortion correction can also be performed on the motion distortion caused by the movement of the target object in the target point cloud data, so as to further improve the accuracy of the dense point cloud data. Specifically, the point cloud clusters corresponding to different target objects in the target point cloud data are first identified, the displacement information of the point cloud clusters at different moments is obtained, and the motion estimation of the target object is performed according to the displacement information. Afterwards, based on the results of motion estimation, distortion correction is performed on the motion distortion of the point cloud clusters induced by the motion of the target object. Wherein, the target point cloud data may be the target point cloud data corrected for the above-mentioned motion distortion caused by the movement of the ranging device. In some embodiments, the distortion correction may be performed on the motion distortion caused by the motion of the target object first, and then the distortion correction is performed on the motion distortion caused by the motion of the distance measuring device.

Wherein, the method for detecting the point cloud cluster of the target object includes but not limited to the scene flow algorithm, the target detection algorithm, or the moving target estimation algorithm based on clustering and segmentation, etc. Of course, in addition, the embodiment of the present invention can also adopt any other A suitable method is to identify the point cloud clusters belonging to the target object, and the embodiment of the present invention does not limit the specific method for identifying the point cloud clusters. After estimating the motion speed of the target object, the distortion correction is performed on the motion distortion caused by the motion of the target object, that is, the high-precision depth data of the target object can be restored.

Exemplarily, when obtaining the displacement information of the point cloud cluster at different times, the point cloud data within a longer time range can be extracted from the point cloud data, and the point cloud cluster of the moving target can be identified from it for motion estimation. The length of the long-term range is greater than the length of the preset time range, so that more accurate displacement information can be obtained.

Afterwards, based on the results of motion estimation, distortion correction is performed on the motion distortion of the point cloud clusters induced by the motion of the target object. Similar to the above-mentioned distortion correction method for the target point cloud data, first assign corresponding correction coefficients to the point cloud clusters at different times according to the time information of the point cloud clusters, and according to the correction coefficients corresponding to the point cloud clusters at different times and the results of motion estimation Distortion correction is performed on point cloud clusters at different moments.

When determining the correction coefficient, it is also necessary to determine a reference time point within the preset time range, and assign correction coefficients to the point cloud points collected at different times according to the interval between the collection time of each point cloud point in the point cloud cluster and the reference time point, Among them, the selection of the reference time point should be consistent with the reference time point selected when performing distortion correction on the target point cloud data. Specifically, the correction coefficients of all point cloud points in the point cloud cluster can be evenly divided by time, so as to assign corresponding correction coefficients to point cloud points by time. The closer the distance to the reference time point, the closer the correction coefficient is to 0, and the distance from the reference point The farther the time point is, the closer the correction factor is to 1. After distributing the distortion coefficients, the motion interpolation estimation at the corresponding moment is applied to obtain the final distortion correction result.

Exemplarily, referring to Fig. 7A and Fig. 7B, taking a vehicle as an example, Fig. 7A is the point cloud cluster of the vehicle before the distortion correction is performed on the motion distortion caused by the movement of the vehicle; Fig. 7B is the point cloud cluster after the distortion correction is performed on Fig. 7A Point cloud clusters of vehicles. From the comparison of FIG. 7A and FIG. 7B , it can be seen that correcting the motion distortion of the point cloud cluster can significantly improve the smear phenomenon of the point cloud cluster.

Since the embodiment of the present invention uses multiple distance measuring devices to jointly collect point cloud data, and the multiple distance measuring devices have the same orientation and tight layout, when there is an object with high reflectivity in the field of view, each distance measuring device may receive The reflected light of laser pulses emitted by other ranging devices forms noise points, which will have a non-negligible impact on the dense point cloud true value data. Therefore, in step S450, identify noise points in the target point cloud data caused by crosstalk between different distance measuring devices, and filter out these noise points, so as to improve the accuracy of the dense point cloud true value data.

Exemplarily, the noise filtering of the point cloud data may be performed using a neighbor point noise filtering method or a depth map noise filtering method. For example, if the number of adjacent points within a certain neighborhood of a certain point cloud point is less than the preset threshold, the point cloud point is an isolated point. Due to the spatial continuity of the real measurement point, the isolated point is usually a noise point, so this can be Some point cloud points are filtered out as noise. In one embodiment, all point cloud points may use the same neighborhood range and preset threshold. In other embodiments, since the point clouds collected for objects that are closer to the distance are relatively dense, and the point clouds collected for objects that are far away are relatively sparse, they are more likely to be judged as noise points. The point cloud points can adopt a larger neighborhood range or a smaller threshold.

Of course, the noise filtering process in step S450 is not limited to the above-mentioned noise filtering method, and the filtered noise is not limited to the noise caused by the crosstalk between different ranging devices, but also includes other types of noise, such as sunlight, dust and ranging Noise caused by reflected light inside the device, etc.

Through the above steps, high-precision dense point cloud data is finally obtained. Afterwards, in step S460, the dense point cloud true value data is output, the dense point cloud true value data includes the target point cloud data after distortion correction and noise filtering processing and the image data after spatial alignment, the two have matching features, Together constitute the dense point cloud ground truth data, as shown in Figure 8. Wherein, the image data may be image data corresponding to a reference time point of distortion correction. After the dense point cloud true value data is obtained, it can be used as a training sample of the deep learning model to train a network model that can generate dense point cloud data based on dense images and sparse point cloud data.

To sum up, the method 400 for constructing dense point cloud true value data according to the embodiment of the present invention uses point cloud data collected by multiple distance measuring devices to construct a dense point cloud, which can avoid the problems of inconsistency in depth and blurring of moving objects caused by inconsistent viewing angles; By performing time alignment, spatial alignment and noise filtering on the point cloud data collected by multiple ranging devices, the accuracy of the dense point cloud true value data is improved.

Next, with reference to FIG. 5 , a system 500 for constructing dense point cloud true-value data according to an embodiment of the present invention will be described, wherein the features of the above-mentioned method 400 for constructing dense point cloud true-value data can be combined into this embodiment.

As shown in Figure 5, the construction system 500 of dense point cloud true value data includes a camera 510, at least two distance measuring devices 520, an inertial measurement device 530 and a processor 550, the camera 510, at least two distance measuring devices 520 and an inertial measurement device The device 530 is set on the same movable platform, and the fields of view of the camera 510 and at least two distance measuring devices 520 overlap at least in part; The point cloud data of the camera 510 and the pose data of the camera 510 and the distance measuring device 520 collected by the inertial measurement device, the field of view of the camera 510 and at least two distance measuring devices 520 at least partially overlap; acquire image data, point cloud data and pose The time information of the data, according to the time information, the image data, point cloud data and pose data are time-aligned; the camera 510, the distance measuring device 520 and the inertial measurement device are calibrated to obtain the image, point cloud data and pose data. The spatial transformation relationship between them, and spatially align the image data, point cloud data and pose data according to the spatial transformation relationship; extract the target point cloud data within the preset time range from the point cloud data transformed into the same space coordinate system According to the change of pose data, the motion distortion in the target point cloud data is distorted to obtain the corrected target point cloud data; the corrected target point cloud data is subjected to noise filtering to obtain the target after noise filtering Point cloud data; output dense point cloud true value data, dense point cloud true value data includes target point cloud data after noise filtering and image data after spatial alignment.

In some embodiments, the orientations of the camera 510 and at least two distance measuring devices 520 are the same. Preferably, the layout of the camera 510 and the at least two distance measuring devices 520 is as close as possible to each other, so as to obtain as large an overlapping field of view as possible, so as to collect point cloud data and image data in the same field of view to the greatest extent.

In some embodiments, the system 500 for constructing dense point cloud true value data further includes a data synchronization system 540, and the data synchronization system 540 is used to: receive image data collected by the camera 510, point cloud data collected by at least two ranging devices 520 , and the pose data collected by the inertial measurement device 530; time synchronization is performed on the image data, point cloud data and pose data; and the time-synchronized image data, point cloud data and pose data are sent to the processor 550.

In some embodiments, the distance measuring device 520 includes a first distance measuring device 520 and at least one second distance measuring device 520, and calibrating the camera 510, the distance measuring device 520 and the inertial measurement device includes: respectively The distance measuring device 520 is calibrated with the first distance measuring device 520 to obtain the space transformation between the second point cloud data collected by each second distance measuring device 520 and the first point cloud data collected by the first distance measuring device 520 Relationship; calibrate the camera 510 and the first distance measuring device 520 to obtain the spatial transformation relationship between the image and the first point cloud data; calibrate the inertial measurement device and the first distance measuring device 520 to obtain the pose data The spatial transformation relationship with the first point cloud data; the spatial alignment includes: transforming the image data, point cloud data and pose data into the space coordinate system of the first point cloud data.

In some embodiments, the distortion correction is performed on the motion distortion in the target point cloud data according to the change of the pose data, including: obtaining the motion direction and speed of the target point cloud data within a preset time range according to the change of the pose data ; Perform distortion correction on the motion distortion caused by the motion of the ranging device 520 in the target point cloud data according to the motion direction and motion speed.

Further, performing distortion correction on the motion distortion caused by the movement of the ranging device 520 in the target point cloud data includes: correcting the target point cloud data to a position corresponding to the initial moment of the preset time range.

Further, performing distortion correction on the motion distortion caused by the movement of the distance measuring device 520 in the target point cloud data includes: assigning corresponding correction coefficients to the target point cloud data at different times according to the time information of the target point cloud data; The correction coefficients, motion directions and speeds corresponding to the target point cloud data at different times are used to correct the distortion of the target point cloud data at different times.

In some embodiments, the processor 550 is also used to: identify point cloud clusters corresponding to different target objects in the target point cloud data; obtain displacement information of the point cloud clusters at different times, and perform motion estimation on the target object according to the displacement information ; Distortion correction is performed on the motion distortion of the point cloud cluster caused by the motion of the target object according to the result of the motion estimation.

In some embodiments, according to the result of the motion estimation, the distortion correction is performed on the motion distortion of the point cloud cluster caused by the motion of the target object, including: assigning corresponding corrections to the point cloud cluster at different times according to the time information of the point cloud cluster Coefficient: Correct the distortion of point cloud clusters at different times according to the correction coefficients corresponding to point cloud clusters at different times and the results of motion estimation.

In some embodiments, the noise filtering process includes: identifying noise points in the target point cloud data caused by crosstalk between different ranging devices 520, and filtering out these noise points.

For more details of the system 500 for constructing dense point cloud true value data in the embodiment of the present invention, reference may be made to the relevant description of the aforementioned method 400 for constructing dense point cloud true value data, and details are not repeated here.

The system 500 for constructing dense point cloud true value data in the embodiment of the present invention uses point cloud data collected by multiple distance measuring devices to construct a dense point cloud, which can avoid the problems of inconsistency in depth and blurring of moving objects caused by inconsistent viewing angles; The point cloud data collected by the ranging device is time-aligned, space-aligned and noise-filtered, which improves the accuracy of the dense point cloud true value data.

Next, an electronic device 900 according to an embodiment of the present invention will be described with reference to FIG. 9 , wherein the features of the aforementioned method 400 for constructing dense point cloud ground truth data can be combined into this embodiment. The electronic device 900 may be implemented as an electronic device such as a computer, a server, or a vehicle-mounted terminal. The electronic device 900 may be carried on a movable platform together with a camera, a distance measuring device, and an inertial measurement device, but is not limited thereto. The mobile platform with the electronic device 900 can measure the external environment, for example, measure the distance between the mobile platform and obstacles for purposes such as obstacle avoidance, and perform two-dimensional or three-dimensional mapping of the external environment.

The electronic device 900 includes one or more processors 920 and one or more memories 910, and the one or more processors 920 work together or individually. Optionally, the electronic device 900 may further include at least one of an input device (not shown), an output device (not shown) and an image sensor (not shown), and these components are connected through a bus system and/or other forms Mechanisms (not shown) are interconnected.

The memory 910 is used for storing processor-executable program instructions, for example, for storing corresponding steps and program instructions for realizing the method for constructing dense point cloud truth data according to the embodiment of the present invention. One or more computer program products may be included, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and/or cache memory (cache). The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, and the like.

The processor 920 may be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other forms of processing with data processing capabilities and/or instruction execution capabilities. unit, and can control other components in the electronic device 900 to perform desired functions. The processor 920 can execute the instructions stored in the memory, so as to execute the method for constructing the dense point cloud ground truth data in the embodiment of the present invention described herein. For example, a processor can include one or more embedded processors, processor cores, microprocessors, logic circuits, hardware finite state machines (FSMs), digital signal processors (DSPs), or combinations thereof. In this embodiment, the processor includes a Field Programmable Gate Array (FPGA), wherein the operation circuit of the ranging device may be a part of the Field Programmable Gate Array (FPGA).

The input device may be a device used by a user to input an instruction, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like. The output device can output various information (such as images or sounds) to the outside (such as a user), and can include one or more of a display, a speaker, etc., and the output device can be used to output dense point cloud true value data.

The communication interface (not shown) is used to communicate with other devices, including wired or wireless communication. The laser ranging device can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 8G, 5G or a combination thereof. In an exemplary embodiment, the communication interface receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication interface further includes a near field communication (NFC) module to facilitate short range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

Specifically, when the program instructions stored in the memory 910 are executed by the processor 920, the processor 920 is used to: acquire the image data collected by the camera, the point cloud data collected by at least two distance measuring devices, and the cameras and data collected by the inertial measurement device. The pose data of the distance measuring device, the camera, at least two distance measuring devices and the inertial measurement device are arranged on the same movable platform, and the fields of view of the camera and the at least two distance measuring devices at least partially overlap; acquire image data, point cloud The time information of data and pose data, time-aligned image data, point cloud data and pose data according to the time information; calibrate the camera, ranging device and inertial measurement device to obtain images, point cloud data and pose The spatial transformation relationship between the data, and spatially align the image data, point cloud data and pose data according to the spatial transformation relationship; extract the target point within the preset time range from the point cloud data transformed into the same space coordinate system Cloud data, according to the change of pose data, the motion distortion in the target point cloud data is distorted to obtain the corrected target point cloud data; the corrected target point cloud data is subjected to noise filtering processing, and the filtered noise processing The target point cloud data; output the dense point cloud true value data, the dense point cloud true value data includes the target point cloud data after noise filtering and the image data after spatial alignment.

The electronic device 900 in the embodiment of the present invention can obtain dense and high-precision dense point cloud truth data.

In addition, an embodiment of the present invention also provides a computer storage medium on which a computer program is stored. One or more computer program instructions can be stored on the computer-readable storage medium, and the processor can execute the program instructions stored in the memory to realize the functions (implemented by the processor) in the embodiments of the present invention described herein And/or other desired functions, such as to execute the corresponding steps of the method for constructing dense point cloud truth data according to the embodiment of the present invention, various application programs and various data can also be stored in the computer-readable storage medium , such as various data used and/or generated by the application.

For example, the computer storage medium may include, for example, a memory card of a smart phone, a memory component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk Read Only Memory (CD-ROM), USB memory, or any combination of the above storage media. The computer readable storage medium can be any combination of one or more computer readable storage medium. Because the computer program instructions stored in the computer storage medium are used to implement the method for constructing the dense point cloud true value data according to the embodiment of the present invention, it also has the above-mentioned advantages.

In the above embodiments, all or part may be implemented by software, hardware, firmware or other arbitrary combinations. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present invention will be generated in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital video disc (digital video disc, DVD)), or a semiconductor medium (such as a solid state disk (solid state disk, SSD)), etc. .

Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above-described example embodiments are exemplary only and are not intended to limit the scope of the invention thereto. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another device, or some features may be omitted, or not implemented.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Similarly, it should be understood that in the description of the exemplary embodiments of the invention, in order to streamline the disclosure and to facilitate an understanding of one or more of the various inventive aspects, various features of the invention are sometimes grouped together in a single embodiment, figure , or in its description. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the corresponding claims reflect, the inventive point lies in that the corresponding technical problem can be solved by using less than all features of a single disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

It will be appreciated by those skilled in the art that all features disclosed in this specification (including accompanying claims, abstract and drawings) and all features of any method or apparatus so disclosed may be used in any combination, except where the features are mutually exclusive. process or unit. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will understand that although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some modules according to the embodiments of the present invention. The present invention can also be implemented as an apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program for realizing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.

Claims

A method for constructing dense point cloud true value data, characterized in that the method comprises:

Obtain image data collected by the camera, point cloud data collected by at least two distance measuring devices, and pose data of the camera and the distance measuring device collected by the inertial measurement device, the camera, the at least two distance measuring devices The device and the inertial measurement device are arranged on the same movable platform, and the fields of view of the camera and the at least two distance measuring devices at least partially overlap;

Acquiring time information of the image data, the point cloud data, and the pose data, and time-aligning the image data, the point cloud data, and the pose data according to the time information;

Calibrate the camera, the distance measuring device and the inertial measurement device to obtain the spatial transformation relationship between the image, the point cloud data and the pose data, and according to the spatial transformation relationship spatially aligning the image data, the point cloud data, and the pose data;

Extract the target point cloud data within the preset time range from the point cloud data transformed into the same space coordinate system, and perform distortion correction on the motion distortion in the target point cloud data according to the change of the pose data, so as to obtain The corrected target point cloud data;

Noise filtering is performed on the corrected target point cloud data to obtain the target point cloud data after noise filtering;

Outputting dense point cloud true value data, the dense point cloud true value data includes the target point cloud data after the noise filtering process and the image data after the spatial alignment.
The method of claim 1, wherein the camera and the at least two distance measuring devices are oriented in the same direction.
The method according to claim 1 or 2, wherein the image data, the point cloud data and the pose data are time-synchronized data.
The method according to any one of claims 1-3, wherein the distance measuring device comprises a first distance measuring device and at least one second distance measuring device, and the pair of the camera, the distance measuring device The device and the inertial measurement unit are calibrated, including:

Calibrating each of the second distance measuring devices and the first distance measuring device respectively, so as to obtain the second point cloud data collected by each of the second distance measuring devices and the point cloud data collected by the first distance measuring device The spatial transformation relationship between the first point cloud data;

Calibrate the camera and the first distance measuring device to obtain a spatial transformation relationship between the image and the first point cloud data;

Calibrating the inertial measurement device and the first distance measuring device to obtain a spatial transformation relationship between the pose data and the first point cloud data;

The spatial alignment includes: transforming the image data, the point cloud data and the pose data into a space coordinate system of the first point cloud data.
The method according to any one of claims 1-4, wherein said performing distortion correction on the motion distortion in the target point cloud data according to the change of the pose data comprises:

Obtaining the movement direction and movement speed of the target point cloud data within the preset time range according to the change of the pose data;

Distortion correction is performed on the motion distortion caused by the motion of the distance measuring device in the target point cloud data according to the motion direction and motion speed.
The method according to claim 5, wherein said performing distortion correction on the motion distortion caused by the motion of the distance measuring device in the target point cloud data comprises:

Correcting the target point cloud data to the position corresponding to the starting moment of the preset time range.
The method according to claim 5 or 6, wherein the distortion correction of the motion distortion caused by the motion of the distance measuring device in the target point cloud data includes:

According to the time information of the target point cloud data, assign corresponding correction coefficients to the target point cloud data at different times;

Distortion correction is performed on the target point cloud data at different times according to the correction coefficients corresponding to the target point cloud data at different times, the moving direction and the moving speed.
The method according to any one of claims 1-7, further comprising:

identifying point cloud clusters corresponding to different target objects in the target point cloud data;

Acquiring displacement information of the point cloud cluster at different times, and performing motion estimation on the target object according to the displacement information;

Distortion correction is performed on the motion distortion of the point cloud cluster caused by the motion of the target object according to the motion estimation result.
The method according to claim 8, wherein, according to the result of the motion estimation, performing distortion correction on the motion distortion of the point cloud cluster caused by the motion of the target object comprises:

Assigning corresponding correction coefficients to point cloud clusters at different times according to the time information of the point cloud clusters;

Distortion correction is performed on the point cloud clusters at different times according to the correction coefficients corresponding to the point cloud clusters at different times and the result of the motion estimation.
The method according to any one of claims 1-9, wherein the noise filtering process comprises: identifying noise points caused by crosstalk between different ranging devices in the target point cloud data, and filtering out the noise.
A system for constructing dense point cloud true value data, characterized in that the system includes a camera, at least two distance measuring devices, an inertial measurement device and a processor, the camera, the at least two distance measuring devices and the The inertial measurement device is arranged on the same movable platform, and the fields of view of the camera and the at least two distance measuring devices at least partially overlap;

The processor is used to:

Acquiring the image data collected by the camera, the point cloud data collected by the at least two distance measuring devices, and the pose data of the camera and the distance measuring device collected by the inertial measurement device, the camera and the distance measuring device The fields of view of the at least two ranging devices at least partially overlap;

Acquiring time information of the image data, the point cloud data, and the pose data, and time-aligning the image data, the point cloud data, and the pose data according to the time information;

Calibrate the camera, the distance measuring device and the inertial measurement device to obtain the spatial transformation relationship between the image, the point cloud data and the pose data, and according to the spatial transformation relationship spatially aligning the image data, the point cloud data, and the pose data;

Extract the target point cloud data within the preset time range from the point cloud data transformed into the same space coordinate system, and perform distortion correction on the motion distortion in the target point cloud data according to the change of the pose data, so as to obtain The corrected target point cloud data;

Noise filtering is performed on the corrected target point cloud data to obtain the target point cloud data after noise filtering;

Outputting dense point cloud true value data, the dense point cloud true value data includes the target point cloud data after the noise filtering process and the image data after the spatial alignment.
The system of claim 11, wherein the camera and the at least two distance measuring devices are oriented in the same direction.
The system according to claim 11 or 12, further comprising a data synchronization system, the data synchronization system is used for:

receiving image data collected by the camera, point cloud data collected by the at least two ranging devices, and pose data collected by the inertial measurement device;

performing time synchronization on the image data, the point cloud data and the pose data;

sending the time-synchronized image data, point cloud data, and pose data to the processor.
The system according to any one of claims 11-13, wherein the distance measuring device comprises a first distance measuring device and at least one second distance measuring device, and the pair of the camera, the distance measuring device The device and the inertial measurement unit are calibrated, including:

Calibrating each of the second distance measuring devices and the first distance measuring device respectively, so as to obtain the second point cloud data collected by each of the second distance measuring devices and the point cloud data collected by the first distance measuring device The spatial transformation relationship between the first point cloud data;

Calibrate the camera and the first distance measuring device to obtain a spatial transformation relationship between the image and the first point cloud data;

Calibrating the inertial measurement device and the first ranging device to obtain a spatial transformation relationship between the pose data and the first point cloud data;

The spatial alignment includes: transforming the image data, the point cloud data and the pose data into a space coordinate system of the first point cloud data.
The system according to any one of claims 11-14, wherein said performing distortion correction on the motion distortion in the target point cloud data according to the change of the pose data comprises:

Obtaining the movement direction and movement speed of the target point cloud data within the preset time range according to the change of the pose data;

Distortion correction is performed on the motion distortion caused by the motion of the distance measuring device in the target point cloud data according to the motion direction and motion speed.
The system according to claim 15, wherein the distortion correction of the motion distortion caused by the motion of the distance measuring device in the target point cloud data comprises:

Correcting the target point cloud data to the position corresponding to the starting moment of the preset time range.
The system according to claim 15 or 16, wherein the distortion correction of the motion distortion caused by the motion of the distance measuring device in the target point cloud data includes:

According to the time information of the target point cloud data, assign corresponding correction coefficients to the target point cloud data at different times;

Distortion correction is performed on the target point cloud data at different times according to the correction coefficients corresponding to the target point cloud data at different times, the moving direction and the moving speed.
The system according to any one of claims 11-17, wherein the processor is further configured to:

identifying point cloud clusters corresponding to different target objects in the target point cloud data;

Obtain the displacement information of the point cloud cluster at different moments, and carry out motion estimation to the target object according to the displacement information;

Distortion correction is performed on the motion distortion of the point cloud cluster caused by the motion of the target object according to the motion estimation result.
The system according to claim 18, wherein said performing distortion correction on the motion distortion of the point cloud cluster caused by the motion of the target object according to the result of the motion estimation comprises:

Assigning corresponding correction coefficients to point cloud clusters at different times according to the time information of the point cloud clusters;

Distortion correction is performed on the point cloud clusters at different times according to the correction coefficients corresponding to the point cloud clusters at different times and the result of the motion estimation.
The system according to any one of claims 11-19, wherein the noise filtering process includes: identifying noise points in the target point cloud data caused by crosstalk between different ranging devices, and filtering out the noise.
An electronic device, characterized in that the electronic device comprises:

memory for storing executable instructions;

A processor, configured to execute the instructions stored in the memory, so that the processor executes the method for constructing dense point cloud truth data according to any one of claims 1 to 10.