WO2023197351A1 - 基于图像与激光点云的图像融合方法、装置、设备和介质 - Google Patents

基于图像与激光点云的图像融合方法、装置、设备和介质 Download PDF

Info

Publication number
WO2023197351A1
WO2023197351A1 PCT/CN2022/088255 CN2022088255W WO2023197351A1 WO 2023197351 A1 WO2023197351 A1 WO 2023197351A1 CN 2022088255 W CN2022088255 W CN 2022088255W WO 2023197351 A1 WO2023197351 A1 WO 2023197351A1
Authority
WO
WIPO (PCT)
Prior art keywords
target
image
cloud data
point cloud
channel
Prior art date
Application number
PCT/CN2022/088255
Other languages
English (en)
French (fr)
Inventor
单佳炜
郑睿童
王世玮
沈罗丰
李洪鹏
Original Assignee
探维科技(北京)有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 探维科技(北京)有限公司 filed Critical 探维科技(北京)有限公司
Priority to US18/255,164 priority Critical patent/US11954835B2/en
Publication of WO2023197351A1 publication Critical patent/WO2023197351A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/251—Fusion techniques of input or preprocessed data
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/803—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of input or preprocessed data
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/4808—Evaluating distance, position or velocity data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10032—Satellite or aerial image; Remote sensing
    • G06T2207/10044—Radar image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/56—Particle system, point based geometry or rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/61—Scene description
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Definitions

  • the present disclosure relates to the field of image fusion technology, and in particular to an image fusion method, device, equipment and medium based on images and laser point clouds.
  • SLAM Simultaneous Localization and Mapping
  • image data is mainly obtained through different sensors to perceive the outside world.
  • a single sensor is mostly used to obtain image data.
  • cameras as the mainstream sensor, can obtain high-resolution two-dimensional images, but cannot provide three-dimensional information of the photographed target, while lidar can obtain three-dimensional information of the photographed target.
  • Point cloud information is less affected by external interference.
  • the point cloud information obtained through lidar has a smaller amount of information, lower resolution, and the problem of edge blur.
  • edge detection is performed using deep learning
  • the target depth edge is obtained by combining the initial depth edge of the initialized depth map, and the target depth edge and the initial depth edge are interpolated and filled to solve the problem of edge blur. Problem,increasing the resolution of depth images.
  • an image fusion method, device, equipment and medium based on images and laser point clouds are provided.
  • Embodiments of the present disclosure provide an image fusion method based on images and laser point clouds.
  • the method includes:
  • the first image obtain a target gradient value corresponding to at least one target pixel point included in the first image, wherein the target pixel point is a non-edge pixel point of the first image;
  • the sparse point cloud data is upsampled to obtain dense point cloud data, wherein the target gradient value is a target corresponding to adjacent channels of the sparse point cloud data. Pixel points are determined;
  • a target fusion map is obtained.
  • obtaining a target gradient value corresponding to at least one target pixel included in the first image according to the first image includes:
  • the target gradient value corresponding to the target pixel point is determined.
  • determining the target gradient value corresponding to the target pixel point based on the gradient value corresponding to each pixel point includes:
  • the target gradient value corresponding to the target pixel point is determined based on the current gradient value corresponding to the pixel point.
  • upsampling the sparse point cloud data based on at least one of the target gradient values to obtain dense point cloud data includes:
  • third depth information corresponding to the target channel is obtained, wherein the target channel is based on the previous channel and the subsequent The target pixels between a channel are determined;
  • the dense point cloud data is obtained based on at least one of the first depth information, at least one of the second depth information, and at least one of the third depth information.
  • obtaining the third depth information corresponding to the target channel based on at least one of the target gradient values, the first depth information and the second depth information includes:
  • the third depth information corresponding to the target channel is determined according to the first target gradient mean value, the second target gradient mean value, the first depth information and the second depth information.
  • the target is determined based on the first target gradient mean value, the second target gradient mean value, the first depth information and the second depth information.
  • the third depth information corresponding to the channel includes:
  • the first target gradient mean value, the second target gradient mean value, the first depth information and the second depth information are weighted and summed to determine the third depth information corresponding to the target channel.
  • the target channel is determined based on the target pixel point between the previous channel and the subsequent channel, including:
  • the target pixel points corresponding to the target channel are determined.
  • the embodiment of the present disclosure provides an image fusion device based on images and laser point clouds, including:
  • An acquisition module configured to acquire the first image and sparse point cloud data, wherein the point cloud data included in each channel of the sparse point cloud data corresponds to the pixel points included in the first image respectively, and the sparse point cloud The data is spatially and temporally synchronized with the first image;
  • a target gradient value obtaining module configured to obtain a target gradient value corresponding to at least one target pixel point included in the first image according to the first image, wherein the target pixel point is a non-edge of the first image. pixel;
  • a dense point cloud data obtaining module configured to upsample the sparse point cloud data based on at least one target gradient value to obtain dense point cloud data, wherein the target gradient value is based on the sparse point cloud data The corresponding target pixel points between adjacent channels are determined;
  • a target fusion map obtaining module is used to obtain a target fusion map based on the first image and the dense point cloud data.
  • An embodiment of the present disclosure provides an electronic device, including a memory and a processor.
  • the memory stores a computer program.
  • the processor executes the computer program, the image fusion based on the image and laser point cloud described in the first aspect is implemented. Method steps.
  • Embodiments of the present disclosure provide a computer-readable storage medium on which a computer program is stored.
  • the computer program is executed by a processor, the steps of the image fusion method based on images and laser point clouds described in the first aspect are implemented.
  • Figure 1 is a schematic flow chart of an image fusion method based on images and laser point clouds provided by an embodiment of the present disclosure
  • Figure 2 is a schematic flow chart of another image fusion method based on images and laser point clouds provided by an embodiment of the present disclosure
  • Figure 3 is a schematic flow chart of yet another image fusion method based on images and laser point clouds provided by an embodiment of the present disclosure
  • Figure 4 is a schematic flow chart of another image fusion method based on images and laser point clouds provided by an embodiment of the present disclosure
  • Figure 5 is a schematic flow chart of another image fusion method based on images and laser point clouds provided by an embodiment of the present disclosure
  • Figure 6 is a schematic structural diagram of an image fusion device based on images and laser point clouds provided by an embodiment of the present disclosure.
  • words such as “exemplary” or “for example” mean examples, illustrations or explanations. Any embodiment or design described as “exemplary” or “such as” in the present disclosure is not intended to be construed as preferred or advantageous over other embodiments or designs. To be precise, the use of words such as “exemplary” or “such as” is intended to present relevant concepts in a specific manner. In addition, in the description of the embodiments of the present disclosure, unless otherwise stated, the meaning of "plurality" refers to both one or more than two.
  • SLAM Simultaneous Localization and Mapping
  • image data is mainly obtained through different sensors to perceive the outside world.
  • a single sensor is mostly used to obtain image data.
  • cameras as the mainstream sensor, can obtain high-resolution two-dimensional images, but cannot provide three-dimensional information of the photographed target, while lidar can obtain three-dimensional information of the photographed target.
  • Point cloud information is less affected by external interference.
  • the point cloud information obtained through lidar has a smaller amount of information, lower resolution, and the problem of edge blur.
  • edge detection is performed using deep learning
  • the target depth edge is obtained by combining the initial depth edge of the initialized depth map, and the target depth edge and the initial depth edge are interpolated and filled to solve the problem of edge blur.
  • using existing technology due to the use of deep learning to extract edge information, there are problems such as large amount of calculation and low efficiency.
  • the present disclosure provides an image fusion method based on images and laser point clouds, by acquiring the first image and sparse point cloud data, wherein the point cloud data included in each channel of the sparse point cloud data and the first image include The pixels correspond respectively, and the sparse point cloud data and the first image have spatial and temporal synchronization; according to the first image, the target gradient value corresponding to at least one target pixel included in the first image is obtained, where the target pixel is the non-edge pixels of the first image; based on at least one target gradient value, upsample the sparse point cloud data to obtain dense point cloud data, where the target gradient value is based on the correspondence between adjacent channels of the sparse point cloud data The target pixel points are determined; based on the first image and the dense point cloud data, a target fusion map is obtained.
  • the guidance Upsample the sparse point cloud data to obtain dense point cloud data, and map the dense point cloud data to the first image to obtain the target fusion map, which avoids the use of deep learning to extract edge information in the existing technology and achieves Upsampling sparse point cloud data, thereby reducing the amount of calculation and improving the resolution of point cloud data.
  • Figure 1 is a schematic flow chart of an image fusion method based on images and laser point clouds provided by an embodiment of the present disclosure, which specifically includes the following steps:
  • the point cloud data included in each channel of the sparse point cloud data corresponds to the pixel points included in the first image respectively; the sparse point cloud data and the first image have spatial and temporal synchronization.
  • the above point cloud data refers to a set of vectors with three-dimensional spatial coordinates obtained using lidar. Because the number is large and dense, it is called a point cloud.
  • the point cloud data includes but is not limited to geometric position information, color information, intensity Information, since point cloud data contains spatial coordinates, it is widely used in many fields such as surveying and mapping, electric power, construction, industry, automobiles, games, criminal investigation, etc., but is not limited to this. This disclosure is not specifically limited. Persons in the field It can be set according to the actual situation.
  • the above channels are used to store geometric position information, color information, and intensity information included in point cloud data. Sparse point cloud data means that when using lidar to obtain a set of vectors with three-dimensional spatial coordinates, the point cloud data corresponding to each pixel in the two-dimensional image corresponding to the sparse point cloud data cannot be completely obtained.
  • first image and sparse point cloud data are obtained based on the image detection module and the photoelectric detection module included in the same radar detection system, mainly by triggering the laser emission module and the image detection module at the same time. This achieves time alignment at the acquisition level of the first image and sparse point cloud data, thereby ensuring that the laser and image detect the same object at the same time, thereby achieving time synchronization and spatial synchronization of point cloud data and image data.
  • the laser emission module and the image detection module are triggered at the same time, the image detection module is used to obtain the first image, and the laser emission module is used to obtain sparse point cloud data.
  • the target pixel is a non-edge pixel of the first image
  • the gradient value refers to the change rate of each pixel in the first image in the X and Y directions, consisting of the change of the X axis and the change of the Y axis. It can be understood as the change compared with adjacent pixels, which is equivalent to expressing the edge in the first image through the difference between two adjacent pixels. That is, when there is an edge in the first image, the The pixel corresponding to the edge has a larger gradient value. Therefore, by obtaining the gradient value of each pixel of the first image, edge contour detection is performed, and the target gradient value corresponding to the non-edge pixel is further determined.
  • the target gradient value corresponding to at least one target pixel included in the first image is obtained.
  • the target gradient value corresponding to at least one target pixel included in the first image is obtained.
  • S121 Perform gradient calculation on each pixel included in the first image to obtain the gradient value corresponding to each pixel.
  • the gradient value of each pixel is obtained by performing gradient calculation on each pixel included in the first image.
  • the gradient value for each pixel can be calculated by setting a gradient operator for the pixel neighborhood in the first image.
  • the gradient operator such as Sobel operator (Sobel operator), Laplacian operator ( LaplaceOperator), and uses the regional convolution kernel to perform convolution calculation on each pixel to obtain the gradient value of each pixel, but is not limited to this.
  • This disclosure is not specifically limited, and those in the field can set it according to the actual situation.
  • the gradient value corresponding to each pixel point is obtained by performing gradient calculation on each pixel point in the first image, and further, one or more non-edges are determined according to the gradient value corresponding to each pixel point.
  • the target gradient value corresponding to the pixel is obtained by performing gradient calculation on each pixel point in the first image, and further, one or more non-edges are determined according to the gradient value corresponding to each pixel point.
  • the gradient value corresponding to each pixel Determine the target gradient value corresponding to the target pixel point.
  • the preset gradient value is a parameter value set to determine whether the pixel is a non-edge pixel based on the gradient value corresponding to each pixel in the first image.
  • the preset gradient value the present disclosure There is no specific limit, and those skilled in the art can make specific settings according to actual conditions.
  • the gradient value corresponding to each pixel point is obtained by performing gradient calculation on each pixel point in the first image, and it is determined whether the gradient value corresponding to each pixel point is less than or equal to the preset gradient threshold. When it is determined that the pixel If the gradient value corresponding to the point is less than or equal to the preset gradient threshold, it indicates that the current pixel is a non-edge pixel, thereby obtaining the target gradient value corresponding to the target pixel.
  • the image fusion method based on images and laser point clouds determines the target gradient corresponding to the non-edge pixels by calculating the gradient value of each pixel in the first image and setting an appropriate preset gradient threshold. value to ensure that the sparse point cloud data is upsampled using the target gradient value corresponding to the non-edge pixels in the first image. And because the non-edge pixels belong to the same connected area in the first image, based on the non-edge pixels The corresponding target gradient value upsamples sparse point cloud data, which can improve the accuracy of obtaining dense point cloud data.
  • S14 Based on at least one target gradient value, upsample the sparse point cloud data to obtain dense point cloud data.
  • the target gradient value is determined based on the corresponding target pixel points between adjacent channels of the sparse point cloud data.
  • Sampling refers to resampling a digital signal. The resampled sampling rate is compared with the original sampling rate to obtain the digital signal. If it is greater than the original signal, it is called upsampling, and if it is smaller, it is called downsampling. Among them, upsampling is also called increased sampling or interpolation, and its essence is interpolation or interpolation.
  • sparse point cloud data is upsampled based on at least one target gradient value to obtain dense point cloud data.
  • One implementation method can be yes:
  • S141 Sequentially traverse to obtain the first depth information corresponding to the previous channel included in the sparse point cloud data, and the second depth information corresponding to the subsequent channel, until the latter channel is the last channel of the sparse point cloud data.
  • depth information refers to the number of bits used to store each pixel. It is also used to measure the resolution of an image. It determines the number of colors for each pixel of a color image, or determines the possible gray levels of each pixel of a grayscale image. series.
  • the first depth information corresponding to the previous channel included in the sparse point cloud data, and the second depth information corresponding to the latter channel are obtained in sequence, until the latter channel.
  • all channels included in the sparse point cloud data do not correspond one-to-one to each pixel in the first image, but correspond to some pixels in the first image. It can be understood that, for There may be multiple target pixels between two adjacent channels.
  • S142 Obtain third depth information corresponding to the target channel based on at least one target gradient value, first depth information, and second depth information.
  • the target channel is determined based on the target pixel point between the previous channel and the next channel, and the target channel is the channel obtained after upsampling between the previous channel and the next channel.
  • a way to determine the target channel may be:
  • the target pixel corresponding to the target channel is determined.
  • the above-mentioned first image is an image of 254*254 size.
  • the former channel corresponds to the pixels in the 10th row and 10th column of the first image, and the latter channel is the 10th row in the first image.
  • the number of target pixels in the 10th row between the previous channel and the next channel is counted to be 10.
  • determine the 10th row in the first image is determined.
  • the pixels in the 15th column are target pixels corresponding to the target channel, but are not limited to this. This disclosure is not specifically limited, and those skilled in the art can determine the target channel according to actual conditions.
  • an implementation of S142 may be:
  • S1421 Obtain the target gradient value of at least one target pixel between the previous channel and the target channel, and the target gradient value of at least one target pixel between the next channel and the target channel.
  • the target gradient values corresponding to one or more target pixels that is, non-edge pixels, between the previous channel and the target channel included in the sparse point cloud data, and obtain the sparse point cloud data.
  • target gradient values corresponding to one or more target pixels that is, non-edge pixels, between the latter channel and the target channel.
  • S1422 Calculate the average of the target gradient value of at least one target pixel between the previous channel and the target channel to obtain the first target gradient mean.
  • S1423 Calculate the mean value of the target gradient value of at least one target pixel between the latter channel and the target channel to obtain the second target gradient mean value.
  • S1424 Determine the third depth information corresponding to the target channel based on the first target gradient mean value, the second target gradient mean value, the first depth information, and the second depth information.
  • the mean value of the target gradient values corresponding to one or more non-edge pixel points between the previous channel and the target channel is calculated to determine the first corresponding to the previous channel.
  • the target gradient mean value, and the mean value of the target gradient values corresponding to one or more non-edge pixels between the latter channel and the target channel is calculated to determine the second target gradient mean value corresponding to the latter channel.
  • the third depth information corresponding to the target channel is calculated based on the first target gradient mean and first depth information corresponding to the previous channel and the second target gradient mean and second depth information corresponding to the subsequent channel.
  • an implementation of S1424 may be:
  • the first target gradient mean value, the second target gradient mean value, the first depth information and the second depth information are weighted and summed to determine the third depth information corresponding to the target channel.
  • D n represents the first depth information corresponding to the previous channel
  • Grad n represents the first target gradient mean corresponding to multiple target pixels between the previous channel and the target channel
  • ⁇ represents the weight corresponding to the previous channel
  • D n+1 represents the second depth information corresponding to the latter channel
  • Grad n+1 represents the second target gradient mean corresponding to multiple target pixels between the latter channel and the target channel
  • ⁇ represents the weight corresponding to the previous channel
  • S143 Obtain dense point cloud data based on at least one first depth information, at least one second depth information, and at least one third depth information.
  • a plurality of first depth information, a plurality of second depth information and a plurality of third depth information are obtained in sequence.
  • the plurality of second depth information Depth information and multiple third depth information determine dense point cloud data.
  • the image fusion method based on images and laser point clouds upsamples the sparse point cloud data by obtaining the target gradient values corresponding to the non-edge pixels between adjacent channels of the sparse point cloud data, that is, Interpolating between adjacent channels of sparse point cloud data to obtain dense and regular point cloud data is equivalent to using the gradient values corresponding to the non-edge pixels of the first image to guide the upsampling of the sparse point cloud data. This improves the resolution of dense point cloud data.
  • the obtained dense point cloud data is mapped to the first image to obtain the target fusion map.
  • the image fusion method based on images and laser point clouds obtaineds the first image and sparse point cloud data, where the point cloud data included in each channel of the sparse point cloud data is the same as the point cloud data included in the first image.
  • the pixels correspond respectively, and the sparse point cloud data and the first image have spatial and temporal synchronization; according to the first image, a target gradient value corresponding to at least one target pixel included in the first image is obtained, wherein the target pixel is the Non-edge pixels of the first image; based on at least one target gradient value, upsample the sparse point cloud data to obtain dense point cloud data, where the target gradient value is based on the correspondence between adjacent channels of the sparse point cloud data The target pixel points are determined; based on the first image and the dense point cloud data, a target fusion map is obtained.
  • the guidance Upsample the sparse point cloud data to obtain dense point cloud data, and map the dense point cloud data to the first image to obtain the target fusion map, which avoids the use of deep learning to extract edge information in the existing technology and achieves Upsampling sparse point cloud data, thereby reducing the amount of calculation and improving the resolution of point cloud data.
  • an image fusion device based on images and laser point clouds including: an acquisition module 10, a target gradient value acquisition module 12, a dense point cloud data acquisition module 14 and a target fusion module.
  • the acquisition module 10 is used to acquire the first image and sparse point cloud data, wherein the point cloud data included in each channel of the sparse point cloud data corresponds to the pixel points included in the first image respectively, and the sparse point cloud data corresponds to the pixel points included in the first image.
  • An image has spatial and temporal synchronization;
  • the target gradient value obtaining module 12 is configured to obtain a target gradient value corresponding to at least one target pixel included in the first image according to the first image, where the target pixel is a non-edge pixel of the first image;
  • the dense point cloud data obtaining module 14 is used to upsample the sparse point cloud data based on at least one target gradient value to obtain dense point cloud data, wherein the target gradient value is based on the correspondence between adjacent channels of the sparse point cloud data.
  • the target pixel is determined;
  • the target fusion map obtaining module 16 is used to obtain the target fusion map based on the first image and dense point cloud data.
  • the target gradient value obtaining module 12 is specifically used to perform gradient calculation on each pixel point included in the first image to obtain the gradient value corresponding to each pixel point; according to the corresponding gradient value of each pixel point, The gradient value determines the target gradient value corresponding to the target pixel point.
  • the target gradient value obtaining module 12 is specifically used to determine whether the gradient value corresponding to each pixel is less than or equal to the preset gradient value; when it is determined that the gradient value corresponding to the pixel is less than or equal to When presetting the gradient value, the target gradient value corresponding to the target pixel is determined based on the gradient value corresponding to the current pixel.
  • the dense point cloud data obtaining module 14 is specifically used to sequentially traverse and obtain the first depth information corresponding to the previous channel and the second depth information corresponding to the latter channel included in the sparse point cloud data. , until the latter channel is the last channel of sparse point cloud data; based on at least one target gradient value, the first depth information and the second depth information, the third depth information corresponding to the target channel is obtained, where the target channel is based on the previous The target pixel points between the channel and the subsequent channel are determined; dense point cloud data is obtained based on at least one first depth information, at least one second depth information and at least one third depth information.
  • the dense point cloud data obtaining module 14 is specifically used to obtain the target gradient value of at least one target pixel between the previous channel and the target channel, and the target gradient value between the next channel and the target channel.
  • the target gradient value of at least one target pixel point between the previous channel and the target channel is calculated as an average to obtain the first target gradient mean value; the target gradient value of at least one target pixel point between the previous channel and the target channel is calculated.
  • Third depth information Third depth information.
  • the dense point cloud data obtaining module 14 is specifically also used to perform a weighted sum of the first target gradient mean, the second target gradient mean, the first depth information, and the second depth information, so as to Determine the third depth information corresponding to the target channel.
  • the dense point cloud data obtaining module 14 is specifically used to count the number of target pixels between the previous channel and the next channel; based on the number of target pixels, determine the target The target pixel corresponding to the channel.
  • the acquisition module 10 is used to acquire the first image and sparse point cloud data, wherein the point cloud data included in each channel of the sparse point cloud data corresponds to the pixel points included in the first image respectively, and the sparse points
  • the cloud data and the first image have spatial and temporal synchronization
  • the target gradient value obtaining module 12 is used to obtain a target gradient value corresponding to at least one target pixel included in the first image according to the first image, where the target pixel is Non-edge pixels of the first image
  • the dense point cloud data obtaining module 14 is used to upsample the sparse point cloud data based on at least one target gradient value to obtain dense point cloud data, wherein the target gradient value is based on the sparse point
  • the corresponding target pixel points between adjacent channels of the cloud data are determined
  • the target fusion map obtaining module 16 is used to obtain the target fusion map based on the first image and dense point cloud data.
  • the guidance Upsample the sparse point cloud data to obtain dense point cloud data, and map the dense point cloud data to the first image to obtain the target fusion map, which avoids the use of deep learning to extract edge information in the existing technology and achieves Upsampling sparse point cloud data, thereby reducing the amount of calculation and improving the resolution of point cloud data.
  • Each module in the above server can be implemented in whole or in part by software, hardware and combinations thereof.
  • Each of the above modules may be embedded in or independent of the processor of the computer device in the form of hardware, or may be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
  • Embodiments of the present disclosure provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor.
  • the processor executes the computer program, it can implement the image-based and For the laser point cloud image fusion method, for example, when the processor executes the computer program, the technical solution of any of the method embodiments shown in Figures 1 to 5 can be implemented. The implementation principles and technical effects are similar and will not be described again here.
  • the present disclosure also provides a computer-readable storage medium on which a computer program is stored.
  • the computer program is executed by a processor, the image fusion method based on images and laser point clouds provided by the embodiments of the present disclosure can be implemented.
  • a computer When the program is executed by the processor, the technical solution of any of the method embodiments shown in Figures 1 to 5 is implemented. The implementation principles and technical effects are similar and will not be described again here.
  • Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc.
  • Volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory.
  • RAM Random Access Memory
  • SRAM Static Random Access Memory
  • DRAM Dynamic Random Access Memory
  • An image fusion method, device, equipment and medium based on images and laser point clouds provided by embodiments of the present disclosure adopt this method to obtain the first image and sparse point cloud data, wherein each channel of the sparse point cloud data
  • the included point cloud data corresponds to the pixel points included in the first image respectively, and the sparse point cloud data and the first image have spatial and temporal synchronization; according to the first image, a target corresponding to at least one target pixel point included in the first image is obtained.
  • the target pixel is a non-edge pixel of the first image; based on at least one target gradient value, the sparse point cloud data is upsampled to obtain dense point cloud data, where the target gradient value is based on the sparse The corresponding target pixel points between adjacent channels of the point cloud data are determined; based on the first image and the dense point cloud data, a target fusion map is obtained.
  • the guidance Upsample the sparse point cloud data to obtain dense point cloud data, and map the dense point cloud data to the first image to obtain the target fusion map, which avoids the use of deep learning to extract edge information in the existing technology and achieves Upsampling sparse point cloud data, thereby reducing the amount of calculation and improving the resolution of point cloud data.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Electromagnetism (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

一种基于图像与激光点云的图像融合方法，包括：获取第一图像以及稀疏点云数据（S10），其中，稀疏点云数据的每个通道包括的点云数据与第一图像包括的像素点分别对应，稀疏点云数据与第一图像具有空间以及时间同步性；根据第一图像，得到第一图像包括的至少一个目标像素点对应的目标梯度值（S12），其中，目标像素点为第一图像的非边缘像素点；基于至少一个目标梯度值，对稀疏点云数据进行上采样，得到稠密点云数据（S14），其中，目标梯度值是根据稀疏点云数据的相邻通道之间对应的目标像素点确定的；基于第一图像以及稠密点云数据，得到目标融合图（S16）。采用该方法能够减少计算量，提高点云数据的分辨率。

Description

基于图像与激光点云的图像融合方法、装置、设备和介质
本公开要求于2022年04月11日提交中国专利局、申请号为202210375538.5、发明名称为“基于图像与激光点云的图像融合方法、装置、设备和介质”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本公开涉及图像融合技术领域，特别是涉及一种基于图像与激光点云的图像融合方法、装置、设备和介质。
背景技术
近年来，在无人驾驶、同步定位与建图(Simultaneous Localization and Mapping，SLAM)等应用领域中，主要通过不同的传感器获取图像数据，以此感知外界的世界。目前，多采用单一的传感器获取图像数据，其中，相机作为主流的传感器，能够获取分辨率较高的二维图像，但是无法提供被拍摄目标的三维信息，而激光雷达能够获取被拍摄目标的三维点云信息，且受外界干扰影响较小，但是，通过激光雷达获得的点云信息相对于二维图像来说，信息量较小，且分辨率较低，存在边缘模糊的问题。
基于此，现有技术中，通过利用深度学习的方式进行边缘检测，结合初始化深度图的初始深度边缘获取目标深度边缘，并对目标深度边缘与初始深度边缘进行插值填充，以此解决边缘模糊的问题，提高深度图像的分辨率。
然而，采用现有技术，由于利用深度学习提取边缘信息，因此存在计算量大、效率低的问题。
发明内容
(一)要解决的技术问题
由于利用深度学习提取边缘信息，因此存在计算量大、效率低的问题。
(二)技术方案
根据本公开公开的各种实施例，提提供了一种基于图像与激光点 云的图像融合方法、装置、设备和介质。
本公开实施例提供了一种基于图像与激光点云的图像融合方法，所述方法包括：
获取第一图像以及稀疏点云数据，其中，所述稀疏点云数据的每个通道包括的点云数据与所述第一图像包括的像素点分别对应，所述稀疏点云数据与所述第一图像具有空间以及时间同步性；
根据所述第一图像，得到所述第一图像包括的至少一个目标像素点对应的目标梯度值，其中，所述目标像素点为所述第一图像的非边缘像素点；
基于至少一个所述目标梯度值，对所述稀疏点云数据进行上采样，得到稠密点云数据，其中，所述目标梯度值是根据所述稀疏点云数据的相邻通道之间对应的目标像素点确定的；
基于所述第一图像以及所述稠密点云数据，得到目标融合图。
作为本公开实施例一种可选的实施方式，所述根据所述第一图像，得到所述第一图像包括的至少一个目标像素点对应的目标梯度值，包括：
对所述第一图像包括的每个像素点进行梯度计算，得到每个像素点对应的梯度值；
根据所述每个像素点对应的梯度值，确定所述目标像素点对应的目标梯度值。
作为本公开实施例一种可选的实施方式，所述根据所述每个像素点对应的梯度值，确定所述目标像素点对应的目标梯度值，包括：
判断所述每个像素点对应的梯度值是否小于或等于预设梯度值；
当确定所述像素点对应的梯度值小于或等于所述预设梯度值时，根据当前所述像素点对应的梯度值，确定所述目标像素点对应的目标梯度值。
作为本公开实施例一种可选的实施方式，所述基于至少一个所述目标梯度值，对所述稀疏点云数据进行上采样，得到稠密点云数据，包括：
依次遍历获取所述稀疏点云数据包括的前一通道对应的第一深度信息、以及后一通道对应的第二深度信息，直至所述后一通道为所述稀疏点云数据的最后一个通道；
基于至少一个所述目标梯度值、所述第一深度信息以及所述第二深度信息，得到目标通道对应的第三深度信息，其中，所述目标通道是根据所述前一通道与所述后一通道之间的目标像素点确定的；
基于至少一个所述第一深度信息、至少一个所述第二深度信息以及至少一个所述第三深度信息，得到所述稠密点云数据。
作为本公开实施例一种可选的实施方式，所述基于至少一个所述目标梯度值、所述第一深度信息以及第二深度信息，得到目标通道对应的第三深度信息，包括：
获取所述前一通道与所述目标通道之间的至少一个目标像素点的目标梯度值、以及所述后一通道与所述目标通道之间的至少一个目标像素点的目标梯度值；
对所述前一通道与所述目标通道之间的至少一个目标像素点的目标梯度值进行均值计算，得到第一目标梯度均值；
对所述后一通道与所述目标通道之间的至少一个目标像素点的目标梯度值进行均值计算，得到第二目标梯度均值；
根据所述第一目标梯度均值、所述第二目标梯度均值、所述第一深度信息以及所述第二深度信息，确定所述目标通道对应的第三深度信息。
作为本公开实施例一种可选的实施方式，所述根据所述第一目标梯度均值、所述第二目标梯度均值、所述第一深度信息以及所述第二深度信息，确定所述目标通道对应的第三深度信息，包括：
对所述第一目标梯度均值、所述第二目标梯度均值、所述第一深度信息以及所述第二深度信息进行加权求和，以确定所述目标通道对应的第三深度信息。
作为本公开实施例一种可选的实施方式，所述目标通道是根据所述前一通道与所述后一通道之间的目标像素点确定的，包括：
统计所述前一通道与所述后一通道之间的目标像素点的个数；
基于所述目标像素点的个数，确定所述目标通道对应的所述目标像素点。
本公开实施例提供了一种基于图像与激光点云的图像融合装置，包括：
获取模块，用于获取第一图像以及稀疏点云数据，其中，所述稀疏点云数据的每个通道包括的点云数据与所述第一图像包括的像素点分别对应，所述稀疏点云数据与所述第一图像具有空间以及时间同步性；
目标梯度值得到模块，用于根据所述第一图像，得到所述第一图像包括的至少一个目标像素点对应的目标梯度值，其中，所述目标像素点为所述第一图像的非边缘像素点；
稠密点云数据得到模块，用于基于至少一个所述目标梯度值，对所述稀疏点云数据进行上采样，得到稠密点云数据，其中，所述目标梯度值是根据所述稀疏点云数据的相邻通道之间对应的目标像素点确定的；
目标融合图得到模块，用于基于所述第一图像以及所述稠密点云数据，得到目标融合图。
本公开实施例提供了一种电子设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现第一方面所述基于图像与激光点云的图像融合方法的步骤。
本公开实施例提供了一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现第一方面所述基于图像与激光点云的图像融合方法的步骤。
本公开的其他特征和优点将在随后的说明书中阐述，并且，部分地从说明书中变得显而易见，或者通过实施本公开而了解。本公开的目的和其他优点在说明书、权利要求书以及附图中所特别指出的结构来实现和获得，本公开的一个或多个实施例的细节在下面的附图和描述中提出。
为使本公开的上述目的、特征和优点能更明显易懂，下文特举可选实施例，并配合所附附图，作详细说明如下。
附图说明
为了更清楚地说明本公开实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，对于本领域普通技术人员而言，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本公开实施例提供的一种基于图像与激光点云的图像融合方法的流程示意图；
图2为本公开实施例提供的另一种基于图像与激光点云的图像融合方法的流程示意图；
图3为本公开实施例提供的再一种基于图像与激光点云的图像融合方法的流程示意图；
图4为本公开实施例提供的又一种基于图像与激光点云的图像融合方法的流程示意图；
图5为本公开实施例提供的又一种基于图像与激光点云的图像融合方法的流程示意图；
图6为本公开实施例提供的一种基于图像与激光点云的图像融合装置的结构示意图。
具体实施方式
为了能够更清楚地理解本公开的上述目的、特征和优点，下面将对本公开的方案进行进一步描述。需要说明的是，在不冲突的情况下，本公开的实施例及实施例中的特征可以相互组合。
在下面的描述中阐述了很多具体细节以便于充分理解本公开，但本公开还可以采用其他不同于在此描述的方式来实施；显然，说明书中的实施例只是本公开的一部分实施例，而不是全部的实施例。
本公开的说明书和权利要求书中的术语“第一”和“第二”等是用来区别不同的对象，而不是用来描述对象的特定顺序。
在本公开实施例中，“示例性的”或者“例如”等词来表示作例子、例证或说明。本公开实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念，此外，在本公开实施例的描述中，除非另有说明，“多个”的含义是指两个或两个以上。
近年来，在无人驾驶、同步定位与建图(Simultaneous Localization and Mapping，SLAM)等应用领域中，主要通过不同的传感器获取图像数据，以此感知外界的世界。目前，多采用单一的传感器获取图像数据，其中，相机作为主流的传感器，能够获取分辨率较高的二维图像，但是无法提供被拍摄目标的三维信息，而激光雷达能够获取被拍摄目标的三维点云信息，且受外界干扰影响较小，但是，通过激光雷达获得的点云信息相对于二维图像来说，信息量较小，且分辨率较低，存在边缘模糊的问题。
基于此，现有技术中，通过利用深度学习的方式进行边缘检测，结合初始化深度图的初始深度边缘获取目标深度边缘，并对目标深度边缘与初始深度边缘进行插值填充，以此解决边缘模糊的问题，提高深度图像的分辨率。然而，采用现有技术，由于利用深度学习提取边缘信息，因此存在计算量大、效率低的问题。
因此，本公开提供了一种基于图像与激光点云的图像融合方法，通过获取第一图像以及稀疏点云数据，其中，稀疏点云数据的每个通道包括的点云数据与第一图像包括的像素点分别对应，稀疏点云数据与第一图像具有空间以及时间同步性；根据第一图像，得到第一图像 包括的至少一个目标像素点对应的目标梯度值，其中，目标像素点为所述第一图像的非边缘像素点；基于至少一个目标梯度值，对稀疏点云数据进行上采样，得到稠密点云数据，其中，目标梯度值是根据稀疏点云数据的相邻通道之间对应的目标像素点确定的；基于第一图像以及所述稠密点云数据，得到目标融合图。这样，通过获取具有时间以及空间同步的第一图像和稀疏点云数据，并利用稀疏点云数据的相邻通道之间与第一图像中对应的多个非边缘像素点的目标梯度值，引导对稀疏点云数据进行上采样，以此得到稠密点云数据，并将稠密点云数据映射到第一图像上，以获取目标融合图，避免了现有技术中利用深度学习提取边缘信息，实现对稀疏点云数据的上采样，从而减少了计算量，提高了点云数据的分辨率。
在一个实施例中，如图1所示，图1为本公开实施例提供的一种基于图像与激光点云的图像融合方法的流程示意图，具体包括以下步骤：
S10：获取第一图像以及稀疏点云数据。
其中，稀疏点云数据的每个通道包括的点云数据与第一图像包括的像素点分别对应；稀疏点云数据与第一图像具有空间以及时间同步性。
上述点云数据是指利用激光雷达获取的具有三维空间坐标的一组向量的集合，由于数量大且密集，因此被称为点云，点云数据包括但不限于几何位置信息、颜色信息、强度信息，由于点云数据带有空间坐标，因此被广泛的应用于测绘、电力、建筑、工业、汽车、游戏、刑侦等相当多的领域，但不限于此，本公开不具体限制，本领域人员可根据实际情况设置。上述通道是指用来存储点云数据包括的几何位置信息、颜色信息、强度信息的。稀疏点云数据是指利用激光雷达在获取具有三维空间坐标的一组向量时，并不能完全获取与稀疏点云数据对应的二维图像中的每个像素点对应的点云数据。
需要说明的是，上述第一图像以及稀疏点云数据是基于同一雷达探测系统中包括的图像探测模组以及光电探测模组获取的，主要通过将激光发射模组与图像探测模组同时触发，以此实现第一图像以及稀疏点云数据获取层面上的时间对准，从而保证了同一时刻激光和图像测到同一个物体，进而实现点云数据和图像数据的时间同步以及空间同步。
具体的，同时触发激光发射模组与图像探测模组，利用图像探测模组获取第一图像，利用激光发射模组获取稀疏点云数据。
S12：根据第一图像，得到第一图像包括的至少一个目标像素点对应的目标梯度值。
其中，目标像素点为第一图像的非边缘像素点；梯度值是指针对第一图像中每个像素点在X与Y方向上的变化率，由X轴的变化与Y轴的变化两个分量组成的，可以理解为与相邻像素进行比较的变化，相当于通过两个相邻像素之间的差值，来表示第一图像中的边缘，即当第一图像中存在边缘时，该边缘处对应的像素点具有较大的梯度值，因此，通过获取第一图像的每个像素点梯度值，从而进行边缘轮廓检测，进一步确定为非边缘像素点对应的目标梯度值。
在上述实施例的基础上，在本公开一些实施例中，如图2所示，根据第一图像，得到第一图像包括的至少一个目标像素点对应的目标梯度值，一种可以实现的方式为：
S121：对第一图像包括的每个像素点进行梯度计算，得到每个像素点对应的梯度值。
具体的，通过对第一图像中包括的每个像素点进行梯度计算，以此得到每个像素点的梯度值。
示例性的，对于每个像素点的梯度值计算可以通过采用对第一图像中的像素邻域设置梯度算子，该梯度算子例如索贝尔算子(Sobeloperator)、拉普拉斯算子(LaplaceOperator)，并利用区域卷积核对每个像素点进行卷积计算，以此得到每个像素点的梯度值，但不限于此，本公开不具体限制，本领域人员可根据实际情况设置。
S122：根据每个像素点对应的梯度值，确定目标像素点对应的目标梯度值。
具体的，通过对第一图像中的每个像素点进行梯度计算，以此得到得到每个像素点对应的梯度值，进一步的，根据每个像素点对应的梯度值确定一个或多个非边缘像素点对应的目标梯度值。
在上述实施例的基础上，在本公开一些实施例中，判断每个像素点对应的梯度值是否小于或等于预设梯度值，如图3所示，根据每个像素点对应的梯度值，确定目标像素点对应的目标梯度值，一种可以实现的方式为：
S1221：当确定像素点对应的梯度值小于或等于预设梯度值时，根据当前像素点对应的梯度值，确定目标像素点对应的目标梯度值。
其中，预设梯度值是用来根据第一图像中每个像素点对应的梯度值，判断该像素点是否为非边缘像素点所设置的参数值，对于预设梯度值的取值，本公开不具体限定，本领域技术人员可根据实际情况具 体设置。
具体的，通过对第一图像中每个像素点进行梯度计算，以此得到每个像素点对应的梯度值，判断每个像素点对应的梯度值是否小于或等于预设梯度阈值，当确定像素点对应的梯度值小于或等于预设梯度阈值，则表明当前像素点为非边缘像素点，以此得到目标像素点对应的目标梯度值。
这样，本公开提供的基于图像与激光点云的图像融合方法，通过计算第一图像中每个像素点的梯度值，并设置合适的预设梯度阈值以此确定非边缘像素点对应的目标梯度值，以此保证利用第一图像中的非边缘像素点对应的目标梯度值对稀疏点云数据进行上采样，又因为非边缘像素点属于第一图像中同一联通区域，因此基于非边缘像素点对应的目标梯度值对稀疏点云数据进行上采样，能够提高获取稠密点云数据的准确性。
S14：基于至少一个目标梯度值，对稀疏点云数据进行上采样，得到稠密点云数据。
其中，目标梯度值是根据稀疏点云数据的相邻通道之间对应的目标像素点确定的。采样是指对数字信号进行重采，重采的采样率与原来获得该数字信号的采样率比较，大于原信号的称为上采样，小于的则称为下采样。其中，上采样也称增取样或者是内插，其实质也就是内插或插值。
在上述实施例的基础上，在本公开的一些实施例中，如图4所示，基于至少一个目标梯度值，对稀疏点云数据进行上采样，得到稠密点云数据，一种实现方式可以是：
S141：依次遍历获取稀疏点云数据包括的前一通道对应的第一深度信息、以及后一通道对应的第二深度信息，直至后一通道为稀疏点云数据的最后一个通道。
其中，深度信息是指存储每个像素所用的位数，它也是用来度量图像的分辨率，其决定彩色图像的每个像素颜色数，或者确定灰度图像的每个像素可能有的灰度级数。
具体的，对于光电探测模组获取的获取的稀疏点云数据，依次获取稀疏点云数据包括的前一通道对应的第一深度信息，以及后一通道对应的第二深度信息，直至后一通道为稀疏点云数据包括的所有通道的最后一个通道。
需要说明的是，对于稀疏点云数据包括的所有通道并不是与第一图像中的每个像素点一一对应，而是与第一图像中的部分像素点分别 对应，可以理解的是，对于相邻的两个通道之间可能存在多个目标像素点。
S142：基于至少一个目标梯度值、第一深度信息以及第二深度信息，得到目标通道对应的第三深度信息。
其中，目标通道是根据前一通道与后一通道之间的目标像素点确定的，目标通道为前一通道与后一通道之间进行上采样之后所得到的通道。
可选的，在上述实施例的基础上，在本公开的一些实施例中，目标通道的一种确定方式可以是：
统计前一通道与后一通道之间的目标像素点的个数。
基于目标像素点的个数，确定目标通道对应的目标像素点。
示例性的，上述第一图像为254*254大小的图像，对于前一通道是与第一图像中的第10行第10列像素点对应的，后一通道为第一图像中的第10行第20列像素点对应的，则统计前一通道与后一通道之间在第10行的目标像素点个数为10，则根据目标像素点个数，确定第第一图像中的第10行第15列像素点为目标通道对应的目标像素点，但不限于此，本公开不具体限制，本领域技术人员可根据实际情况确定目标通道。
进一步的，在上述实施例的基础上，在本公开的一些实施例中，如图5所示，S142的一种实现方式可以是：
S1421：获取前一通道与目标通道之间的至少一个目标像素点的目标梯度值、以及后一通道与目标通道之间的至少一个目标像素点的目标梯度值。
具体的，当确定目标通道之后，获取稀疏点云数据包括的前一通道与目标通道之间的一个或多个目标像素点即非边缘像素点分别对应的目标梯度值，以及获取稀疏点云数据包括的后一通道与目标通道之间的一个或多个目标像素点即非边缘像素点分别对应的目标梯度值。
S1422：对前一通道与目标通道之间的至少一个目标像素点的目标梯度值进行均值计算，得到第一目标梯度均值。
S1423：对后一通道与目标通道之间的至少一个目标像素点的目标梯度值进行均值计算，得到第二目标梯度均值。
S1424：根据第一目标梯度均值、第二目标梯度均值、第一深度信息以及所述第二深度信息，确定目标通道对应的第三深度信息。
具体的，当获取的前一通道与目标通道之间的一个或多个目标像素点即非边缘像素点分别对应的目标梯度值，以及后一通道与目标通 道之间的一个或多个目标像素点即非边缘像素点分别对应的目标梯度值之后，分别计算前一通道与目标通道之间的一个或多个非边缘像素点对应的目标梯度值的均值，以确定前一通道对应的第一目标梯度均值，以及后一通道与目标通道之间的一个或多个非边缘像素点对应的目标梯度值的均值，以确定后一通道对应的第二目标梯度均值。进一步的，根据前一通道对应的第一目标梯度均值、第一深度信息以及后一通道对应的第二目标梯度均值、第二深度信息，计算得到目标通道对应的第三深度信息。
可选的，在上述实施例的基础上，在本公开的一些实施例中，S1424的一种实现方式可以是：
对第一目标梯度均值、第二目标梯度均值、第一深度信息以及第二深度信息进行加权求和，以确定目标通道对应的第三深度信息。
具体的，根据公式D n′＝αD nGrad n+βD n+1Grad n+1确定目标通道对应的第三深度信息。
其中，D n表示前一通道对应的第一深度信息，Grad n表示前一通道与目标通道之间的多个目标像素点对应的第一目标梯度均值，α表示前一通道对应的权重，D n+1表示后一通道对应的第二深度信息，Grad n+1表示后一通道与目标通道之间的多个目标像素点对应的第二目标梯度均值，β表示前一通道对应的权重，对于α以及β的取值，本公开不具体限制，本领域技术人员可根据实际情况设置。
S143：基于至少一个第一深度信息、至少一个第二深度信息以及至少一个第三深度信息，得到稠密点云数据。
具体的，根据稀疏点云数据中包括的所有通道，依次得到多个第一深度信息、多个第二深度信息以及多个第三深度信息，基于该多个第一深度信息、多个第二深度信息以及多个第三深度信息，确定稠密点云数据。
这样，本实施例提供的基于图像与激光点云的图像融合方法，通过获取稀疏点云数据相邻通道之间的非边缘像素点对应的目标梯度值，对稀疏点云数据进行上采样，即在稀疏点云数据相邻通道之间进行插值，以此得到稠密有规则的点云数据，相当于利用第一图像的非边缘像素点对应的梯度值引导对稀疏点云数据的上采样，以此提高了稠密点云数据的分辨率。
S16:基于第一图像以及稠密点云数据，得到目标融合图。
具体的，将得到的稠密点云数据映射到第一图像中，以此得到目 标融合图。
这样，本实施例提供的基于图像与激光点云的图像融合方法，通过获取第一图像以及稀疏点云数据，其中，稀疏点云数据的每个通道包括的点云数据与第一图像包括的像素点分别对应，稀疏点云数据与第一图像具有空间以及时间同步性；根据第一图像，得到第一图像包括的至少一个目标像素点对应的目标梯度值，其中，目标像素点为所述第一图像的非边缘像素点；基于至少一个目标梯度值，对稀疏点云数据进行上采样，得到稠密点云数据，其中，目标梯度值是根据稀疏点云数据的相邻通道之间对应的目标像素点确定的；基于第一图像以及所述稠密点云数据，得到目标融合图。这样，通过获取具有时间以及空间同步的第一图像和稀疏点云数据，并利用稀疏点云数据的相邻通道之间与第一图像中对应的多个非边缘像素点的目标梯度值，引导对稀疏点云数据进行上采样，以此得到稠密点云数据，并将稠密点云数据映射到第一图像上，以获取目标融合图，避免了现有技术中利用深度学习提取边缘信息，实现对稀疏点云数据的上采样，从而减少了计算量，提高了点云数据的分辨率。
应该理解的是，虽然图1-图5的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，图1-图5中的至少一部分步骤可以包括多个子步骤或者多个阶段，这些子步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些子步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤的子步骤或者阶段的至少一部分轮流或者交替地执行。
在一个实施例中，如图6所示，提供了一种基于图像与激光点云的图像融合装置，包括：获取模块10、目标梯度值得到模块12、稠密点云数据得到模块14以及目标融合图得到模块16。
其中，获取模块10，用于获取第一图像以及稀疏点云数据，其中，稀疏点云数据的每个通道包括的点云数据与第一图像包括的像素点分别对应，稀疏点云数据与第一图像具有空间以及时间同步性；
目标梯度值得到模块12，用于根据第一图像，得到第一图像包括的至少一个目标像素点对应的目标梯度值，其中，目标像素点为第一图像的非边缘像素点；
稠密点云数据得到模块14，用于基于至少一个目标梯度值，对稀疏点云数据进行上采样，得到稠密点云数据，其中，目标梯度值是根据稀疏点云数据的相邻通道之间对应的目标像素点确定的；
目标融合图得到模块16，用于基于第一图像以及稠密点云数据，得到目标融合图。
在本发明实施例一实施方式中，目标梯度值得到模块12，具体用于对第一图像包括的每个像素点进行梯度计算，得到每个像素点对应的梯度值；根据每个像素点对应的梯度值，确定目标像素点对应的目标梯度值。
在本发明实施例一实施方式中，目标梯度值得到模块12，具体还用于判断每个像素点对应的梯度值是否小于或等于预设梯度值；当确定像素点对应的梯度值小于或等于预设梯度值时，根据当前像素点对应的梯度值，确定目标像素点对应的目标梯度值。
在本发明实施例一实施方式中，稠密点云数据得到模块14，具体用于依次遍历获取稀疏点云数据包括的前一通道对应的第一深度信息、以及后一通道对应的第二深度信息，直至后一通道为稀疏点云数据的最后一个通道；基于至少一个目标梯度值、第一深度信息以及第二深度信息，得到目标通道对应的第三深度信息，其中，目标通道是根据前一通道与后一通道之间的目标像素点确定的；基于至少一个第一深度信息、至少一个第二深度信息以及至少一个第三深度信息，得到稠密点云数据。
在本发明实施例一实施方式中，稠密点云数据得到模块14，具体还用于获取前一通道与目标通道之间的至少一个目标像素点的目标梯度值、以及后一通道与目标通道之间的至少一个目标像素点的目标梯度值；对前一通道与目标通道之间的至少一个目标像素点的目标梯度值进行均值计算，得到第一目标梯度均值；对后一通道与目标通道之间的至少一个目标像素点的目标梯度值进行均值计算，得到第二目标梯度均值；根据第一目标梯度均值、第二目标梯度均值、第一深度信息以及第二深度信息，确定目标通道对应的第三深度信息。
在本发明实施例一实施方式中，稠密点云数据得到模块14，具体还用于对第一目标梯度均值、第二目标梯度均值、第一深度信息以及第二深度信息进行加权求和，以确定目标通道对应的第三深度信息。
在本发明实施例一实施方式中，稠密点云数据得到模块14，具体 还用于统计前一通道与后一通道之间的目标像素点的个数；基于目标像素点的个数，确定目标通道对应的目标像素点。
在上述实施例中，获取模块10，用于获取第一图像以及稀疏点云数据，其中，稀疏点云数据的每个通道包括的点云数据与第一图像包括的像素点分别对应，稀疏点云数据与第一图像具有空间以及时间同步性；目标梯度值得到模块12，用于根据第一图像，得到第一图像包括的至少一个目标像素点对应的目标梯度值，其中，目标像素点为第一图像的非边缘像素点；稠密点云数据得到模块14，用于基于至少一个目标梯度值，对稀疏点云数据进行上采样，得到稠密点云数据，其中，目标梯度值是根据稀疏点云数据的相邻通道之间对应的目标像素点确定的；目标融合图得到模块16，用于基于第一图像以及稠密点云数据，得到目标融合图。这样，通过获取具有时间以及空间同步的第一图像和稀疏点云数据，并利用稀疏点云数据的相邻通道之间与第一图像中对应的多个非边缘像素点的目标梯度值，引导对稀疏点云数据进行上采样，以此得到稠密点云数据，并将稠密点云数据映射到第一图像上，以获取目标融合图，避免了现有技术中利用深度学习提取边缘信息，实现对稀疏点云数据的上采样，从而减少了计算量，提高了点云数据的分辨率。
关于基于图像与激光点云的图像融合装置的具体限定可以参见上文中对于基于图像与激光点云的图像融合方法的限定，在此不再赘述。上述服务器中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中，也可以以软件形式存储于计算机设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。
本公开实施例提供了一种电子设备，包括：存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，处理器执行计算机程序时可以实现本公开实施例提供的基于图像与激光点云的图像融合方法，例如，处理器执行计算机程序时可以实现图1到图5任一所示方法实施例的技术方案，其实现原理和技术效果类似，此处不再赘述。
本公开还提供一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时可以实现本公开实施例提供的基于图像与激光点云的图像融合方法，例如，计算机程序被处理器执行时实现图1到图5任一所示方法实施例的技术方案，其实现原理和技术效果类似，此处不再赘述。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部 分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一非易失性计算机可读取存储介质中，该计算机程序在执行时，可包括如上述各方法的实施例的流程。其中，本公开所提供的各实施例中所使用的对存储器、数据库或其它介质的任何引用，均可包括非易失性和易失性存储器中的至少一种。非易失性存储器可包括只读存储器(Read-Only Memory，ROM)、磁带、软盘、闪存或光存储器等。易失性存储器可包括随机存取存储器(Random Access Memory，RAM)或者外部高速缓冲存储器。作为说明而非局限，RAM以多种形式可得，比如静态随机存取存储器(Static Random Access Memory，SRAM)和动态随机存取存储器(Dynamic Random Access Memory，DRAM)等。
以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上所述实施例仅表达了本公开的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本公开构思的前提下，还可以做出若干变形和改进，这些都属于本公开的保护范围。因此，本公开专利的保护范围应以所附权利要求为准。
工业实用性
本公开实施例所提供的一种基于图像与激光点云的图像融合方法、装置、设备和介质，采用该方式通过获取第一图像以及稀疏点云数据，其中，稀疏点云数据的每个通道包括的点云数据与第一图像包括的像素点分别对应，稀疏点云数据与第一图像具有空间以及时间同步性；根据第一图像，得到第一图像包括的至少一个目标像素点对应的目标梯度值，其中，目标像素点为所述第一图像的非边缘像素点；基于至少一个目标梯度值，对稀疏点云数据进行上采样，得到稠密点云数据，其中，目标梯度值是根据稀疏点云数据的相邻通道之间对应的目标像素点确定的；基于第一图像以及所述稠密点云数据，得到目标融合图。这样，通过获取具有时间以及空间同步的第一图像和稀疏点云数据，并利用稀疏点云数据的相邻通道之间与第一图像中对应的多个非边缘像素点的目标梯度值，引导对稀疏点云数据进行上采样，以此得到稠密点云数据，并将稠密点云数据映射到第一图像上，以获取目标融合 图，避免了现有技术中利用深度学习提取边缘信息，实现对稀疏点云数据的上采样，从而减少了计算量，提高了点云数据的分辨率。

Claims (10)

  1. 一种基于图像与激光点云的图像融合方法，其特征在于，包括：
    获取第一图像以及稀疏点云数据，其中，所述稀疏点云数据的每个通道包括的点云数据与所述第一图像包括的像素点分别对应，所述稀疏点云数据与所述第一图像具有空间以及时间同步性；
    根据所述第一图像，得到所述第一图像包括的至少一个目标像素点对应的目标梯度值，其中，所述目标像素点为所述第一图像的非边缘像素点；
    基于至少一个所述目标梯度值，对所述稀疏点云数据进行上采样，得到稠密点云数据，其中，所述目标梯度值是根据所述稀疏点云数据的相邻通道之间对应的目标像素点确定的；
    基于所述第一图像以及所述稠密点云数据，得到目标融合图。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述第一图像，得到所述第一图像包括的至少一个目标像素点对应的目标梯度值，包括：
    对所述第一图像包括的每个像素点进行梯度计算，得到每个像素点对应的梯度值；
    根据所述每个像素点对应的梯度值，确定所述目标像素点对应的目标梯度值。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述每个像素点对应的梯度值，确定所述目标像素点对应的目标梯度值，包括：
    判断所述每个像素点对应的梯度值是否小于或等于预设梯度值；
    当确定所述像素点对应的梯度值小于或等于所述预设梯度值时，根据当前所述像素点对应的梯度值，确定所述目标像素点对应的目标梯度值。
  4. 根据权利要求1所述的方法，其特征在于，所述基于至少一个所述目标梯度值，对所述稀疏点云数据进行上采样，得到稠密点云数据，包括：
    依次遍历获取所述稀疏点云数据包括的前一通道对应的第一深度信息、以及后一通道对应的第二深度信息，直至所述后一通道为所述稀疏点云数据的最后一个通道；
    基于至少一个所述目标梯度值、所述第一深度信息以及所述第二深度信息，得到目标通道对应的第三深度信息，其中，所述目标通道 是根据所述前一通道与所述后一通道之间的目标像素点确定的；
    基于至少一个所述第一深度信息、至少一个所述第二深度信息以及至少一个所述第三深度信息，得到所述稠密点云数据。
  5. 根据权利要求4所述的方法，其特征在于，所述基于至少一个所述目标梯度值、所述第一深度信息以及第二深度信息，得到目标通道对应的第三深度信息，包括：
    获取所述前一通道与所述目标通道之间的至少一个目标像素点的目标梯度值、以及所述后一通道与所述目标通道之间的至少一个目标像素点的目标梯度值；
    对所述前一通道与所述目标通道之间的至少一个目标像素点的目标梯度值进行均值计算，得到第一目标梯度均值；
    对所述后一通道与所述目标通道之间的至少一个目标像素点的目标梯度值进行均值计算，得到第二目标梯度均值；
    根据所述第一目标梯度均值、所述第二目标梯度均值、所述第一深度信息以及所述第二深度信息，确定所述目标通道对应的第三深度信息。
  6. 根据权利要求5所述的方法，其特征在于，所述根据所述第一目标梯度均值、所述第二目标梯度均值、所述第一深度信息以及所述第二深度信息，确定所述目标通道对应的第三深度信息，包括：
    对所述第一目标梯度均值、所述第二目标梯度均值、所述第一深度信息以及所述第二深度信息进行加权求和，以确定所述目标通道对应的第三深度信息。
  7. 根据权利要求4所述的方法，其特征在于，所述目标通道是根据所述前一通道与所述后一通道之间的目标像素点确定的，包括：
    统计所述前一通道与所述后一通道之间的目标像素点的个数；
    基于所述目标像素点的个数，确定所述目标通道对应的所述目标像素点。
  8. 一种基于图像与激光点云的图像融合装置，其特征在于，包括：
    获取模块，用于获取第一图像以及稀疏点云数据，其中，所述稀疏点云数据的每个通道包括的点云数据与所述第一图像包括的像素点分别对应所述稀疏点云数据与所述第一图像具有空间以及时间同步性；
    目标梯度值得到模块，用于根据所述第一图像，得到所述第一图像包括的至少一个目标像素点对应的目标梯度值，其中，所述目标像素点为所述第一图像的非边缘像素点；
    稠密点云数据得到模块，用于基于至少一个所述目标梯度值，对 所述稀疏点云数据进行上采样，得到稠密点云数据，其中，所述目标梯度值是根据所述稀疏点云数据的相邻通道之间对应的目标像素点确定的；
    目标融合图得到模块，用于基于所述第一图像以及所述稠密点云数据，得到目标融合图。
  9. 一种电子设备，包括存储器和处理器，所述存储器存储有计算机程序，其特征在于，所述处理器执行所述计算机程序时实现权利要求1至7中任一项所述基于图像与激光点云的图像融合方法的步骤。
  10. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现权利要求1-7任一项所述基于图像与激光点云的图像融合方法的步骤。
PCT/CN2022/088255 2022-04-11 2022-04-21 基于图像与激光点云的图像融合方法、装置、设备和介质 WO2023197351A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/255,164 US11954835B2 (en) 2022-04-11 2022-04-21 Methods, devices, apparatuses, and media for image fusion utilizing images and LiDAR point clouds

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210375538.5A CN114677315B (zh) 2022-04-11 2022-04-11 基于图像与激光点云的图像融合方法、装置、设备和介质
CN202210375538.5 2022-04-11

Publications (1)

Publication Number Publication Date
WO2023197351A1 true WO2023197351A1 (zh) 2023-10-19

Family

ID=82078924

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/088255 WO2023197351A1 (zh) 2022-04-11 2022-04-21 基于图像与激光点云的图像融合方法、装置、设备和介质

Country Status (3)

Country Link
US (1) US11954835B2 (zh)
CN (1) CN114677315B (zh)
WO (1) WO2023197351A1 (zh)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107610084A (zh) * 2017-09-30 2018-01-19 驭势科技（北京）有限公司 一种对深度图像和激光点云图进行信息融合的方法与设备
US20180101932A1 (en) * 2016-10-11 2018-04-12 The Boeing Company System and method for upsampling of sparse point cloud for 3d registration
CN111161338A (zh) * 2019-12-26 2020-05-15 浙江大学 一种基于二维图像灰度进行深度预测的点云密度提升方法
CN112085842A (zh) * 2019-06-14 2020-12-15 北京京东尚科信息技术有限公司 深度值确定方法及装置、电子设备和存储介质
CN112150595A (zh) * 2020-09-21 2020-12-29 广东博智林机器人有限公司 一种点云数据处理方法、装置、设备及介质
CN112731436A (zh) * 2020-12-17 2021-04-30 浙江大学 基于点云上采样的多模态数据融合可行驶区域检测方法
CN112766328A (zh) * 2020-01-05 2021-05-07 北京航空航天大学 融合激光雷达、双目相机和ToF深度相机数据的智能机器人深度图像构建方法

Family Cites Families (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10699476B2 (en) * 2015-08-06 2020-06-30 Ams Sensors Singapore Pte. Ltd. Generating a merged, fused three-dimensional point cloud based on captured images of a scene
EP3570253B1 (en) 2017-02-17 2021-03-17 SZ DJI Technology Co., Ltd. Method and device for reconstructing three-dimensional point cloud
DE112018004891T5 (de) * 2017-09-01 2020-06-10 Sony Corporation Bildverarbeitungsvorrichtung, bildverarbeitungsverfahren, programm und mobiler körper
US10628671B2 (en) * 2017-11-01 2020-04-21 Here Global B.V. Road modeling from overhead imagery
CN108061904B (zh) 2017-12-29 2020-12-22 华为技术有限公司 多线激光雷达
EP3818474A4 (en) * 2018-07-05 2022-04-06 Optimum Semiconductor Technologies, Inc. OBJECT DETECTION WITH MULTIPLE SENSORS AND REDUCED COMPLEX NEURAL NETWORKS
CN109300190B (zh) * 2018-09-06 2021-08-10 百度在线网络技术（北京）有限公司 三维数据的处理方法、装置、设备和存储介质
CN111724478B (zh) * 2020-05-19 2021-05-18 华南理工大学 一种基于深度学习的点云上采样方法
CN111563923B (zh) * 2020-07-15 2020-11-10 浙江大华技术股份有限公司 获得稠密深度图的方法及相关装置
RU2745010C1 (ru) * 2020-08-25 2021-03-18 Самсунг Электроникс Ко., Лтд. Способы реконструкции карты глубины и электронное вычислительное устройство для их реализации
CN112102472B (zh) * 2020-09-01 2022-04-29 北京航空航天大学 稀疏三维点云稠密化方法
CN113160327A (zh) * 2021-04-09 2021-07-23 上海智蕙林医疗科技有限公司 一种点云补全的实现方法和系统
CN113160390B (zh) * 2021-04-28 2022-07-22 北京理工大学 一种三维稠密重建方法及系统
CN114140758A (zh) * 2021-11-30 2022-03-04 北京超星未来科技有限公司 一种目标检测方法、装置及计算机设备

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180101932A1 (en) * 2016-10-11 2018-04-12 The Boeing Company System and method for upsampling of sparse point cloud for 3d registration
CN107610084A (zh) * 2017-09-30 2018-01-19 驭势科技（北京）有限公司 一种对深度图像和激光点云图进行信息融合的方法与设备
CN112085842A (zh) * 2019-06-14 2020-12-15 北京京东尚科信息技术有限公司 深度值确定方法及装置、电子设备和存储介质
CN111161338A (zh) * 2019-12-26 2020-05-15 浙江大学 一种基于二维图像灰度进行深度预测的点云密度提升方法
CN112766328A (zh) * 2020-01-05 2021-05-07 北京航空航天大学 融合激光雷达、双目相机和ToF深度相机数据的智能机器人深度图像构建方法
CN112150595A (zh) * 2020-09-21 2020-12-29 广东博智林机器人有限公司 一种点云数据处理方法、装置、设备及介质
CN112731436A (zh) * 2020-12-17 2021-04-30 浙江大学 基于点云上采样的多模态数据融合可行驶区域检测方法

Also Published As

Publication number Publication date
US11954835B2 (en) 2024-04-09
CN114677315A (zh) 2022-06-28
CN114677315B (zh) 2022-11-29
US20240037707A1 (en) 2024-02-01

Similar Documents

Publication Publication Date Title
TWI762860B (zh) 目標檢測及目標檢測網路的訓練方法、裝置、設備及儲存媒體
CN106651938B (zh) 一种融合高分辨率彩色图像的深度图增强方法
US10762655B1 (en) Disparity estimation using sparsely-distributed phase detection pixels
KR101881243B1 (ko) 깊이 이미지를 프로세싱하기 위한 방법 및 장치
JP6201148B2 (ja) キャリブレーション装置、キャリブレーション方法、キャリブレーション機能を備えた移動体搭載用カメラ及びプログラム
CN105051785A (zh) 用于运动估算的方法和装置
CN108521554B (zh) 大场景多目标协同跟踪方法、智能监控系统、交通系统
US11244426B2 (en) Method for image super resolution imitating optical zoom implemented on a resource-constrained mobile device, and a mobile device implementing the same
CN111105347B (zh) 一种生成带深度信息的全景图的方法、装置及存储介质
CN112287824A (zh) 基于双目视觉的三维目标检测方法、装置及系统
CN110738731A (zh) 一种用于双目视觉的3d重建方法和系统
JP6830712B1 (ja) ランダムサンプリング一貫性に基づく魚眼画像の有効領域抽出方法
CN112947419A (zh) 避障方法、装置及设备
US20240073523A1 (en) Systems and methods for generating depth information from low-resolution images
CN113514054A (zh) 一种星敏感器星点像斑检测方法及系统
CN111292369A (zh) 激光雷达的伪点云数据生成方法
CN114419568A (zh) 一种基于特征融合的多视角行人检测方法
CN112017259B (zh) 一种基于深度相机与热像仪的室内定位与建图方法
WO2023197351A1 (zh) 基于图像与激光点云的图像融合方法、装置、设备和介质
JP7315216B2 (ja) 補正距離算出装置、補正距離算出用プログラムおよび補正距離算出方法
CN115546027B (zh) 图像缝合线确定方法、装置以及存储介质
CN116503248A (zh) 一种针对原油储罐的红外图像校正方法及校正系统
CN109242900B (zh) 焦平面定位方法、处理装置、焦平面定位系统及存储介质
JP6991700B2 (ja) 情報処理装置、情報処理方法、プログラム
KR20220133766A (ko) 멀티뷰 어안 렌즈들을 이용한 실시간 전방위 스테레오 매칭 방법 및 그 시스템

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22936995

Country of ref document: EP

Kind code of ref document: A1