CN114627174A - Depth map generation system and method and autonomous mobile device - Google Patents
Depth map generation system and method and autonomous mobile device Download PDFInfo
- Publication number
- CN114627174A CN114627174A CN202210328464.XA CN202210328464A CN114627174A CN 114627174 A CN114627174 A CN 114627174A CN 202210328464 A CN202210328464 A CN 202210328464A CN 114627174 A CN114627174 A CN 114627174A
- Authority
- CN
- China
- Prior art keywords
- baseline
- depth
- pixel
- speckle
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/514—Depth or shape recovery from specularities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/30—Assessment of water resources
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Measurement Of Optical Distance (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
The embodiment of the invention provides a depth map generation system, a depth map generation method and autonomous mobile equipment, which are applied to the technical field of information, and project light rays through a plurality of projectors; a sensor collects an image to be processed; the processor compares the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel of each speckle position in the image to be processed; aiming at any baseline, calculating the depth value of each pixel of each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel of each speckle position; aiming at any baseline, generating a depth map corresponding to the baseline according to the depth value of each pixel of each speckle position corresponding to the baseline; and fusing the depth maps corresponding to the baselines to obtain a target depth map. Therefore, the corresponding depth maps can be respectively calculated by utilizing a plurality of baselines with different distances between a plurality of projectors and the sensors, and the calculated depth maps are fused, so that the accuracy of depth calculation is improved.
Description
Technical Field
The present invention relates to the field of information technologies, and in particular, to a depth map generation system, a depth map generation method, and an autonomous mobile device.
Background
At present, along with the popularization of intelligent equipment, the application of automatic obstacle avoidance equipment is more and more extensive. For example, by sweeping the robot, surrounding obstacles can be identified and avoided. When obstacle recognition is carried out, speckles need to be projected through a projector, then images including the speckles are collected through a sensor, finally, depth calculation is carried out through the collected images, a depth map is generated, and therefore obstacle recognition and avoidance are carried out through the depth map.
In the prior art, in order to improve the accuracy of depth calculation, a plurality of sensors are generally arranged for acquisition, and a three-dimensional distance measurement method is used for measurement by using baselines between different sensors and a projector. However, this method not only has high requirements for chips, but also results in high system complexity.
Disclosure of Invention
The embodiment of the invention aims to provide a depth map generation system and method and autonomous mobile equipment so as to improve the accuracy of depth calculation. The specific technical scheme is as follows:
in a first aspect of embodiments of the present application, there is provided a depth map generating system, including a sensor, a processor, and a plurality of projectors;
the projectors are used for projecting light rays;
the sensor is used for acquiring an image to be processed, wherein the image to be processed comprises speckles formed by reflecting light rays projected by the plurality of projectors;
the processor is used for comparing the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel of each speckle position in the image to be processed, wherein the preset speckle image comprises the initial reference position of each pixel of each speckle position; aiming at any baseline, calculating the depth value of each pixel of each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel of each speckle position, wherein the baseline represents the distance between a projector and the sensor, and the distances between different projectors and the sensor are different; aiming at any baseline, generating a depth map corresponding to the baseline according to the depth value of each pixel of each speckle position corresponding to the baseline; and fusing the depth maps corresponding to the baselines to obtain a target depth map.
Optionally, the processor is specifically configured to, for any baseline, calculate a depth value of each pixel of the plurality of speckle positions according to the baseline and a disparity corresponding to each pixel of each speckle position; and aiming at any baseline, extracting the depth value of the speckles with the depth values within the preset depth range according to the preset depth range corresponding to the baseline to obtain the depth value of each pixel of each speckle position corresponding to the baseline, wherein each baseline corresponds to one preset depth range, and the preset depth ranges of different baselines are different.
Optionally, the processor is specifically configured to select, for any baseline, a plane equation corresponding to the baseline; and calculating the depth of each pixel of each speckle position according to the selected plane equation and the parallax corresponding to each pixel of each speckle position to obtain the depth value of each pixel of each speckle position.
Optionally, the plurality of projectors are specifically configured to alternately project light through each of the projectors;
the sensor is specifically used for acquiring the image to be processed once every preset time interval.
Optionally, an exposure time period of a projector closest to the sensor among the plurality of projectors is shorter than an exposure time period of a projector farthest from the sensor.
In a second aspect of the embodiments of the present application, a depth map generation method is provided, which is applied to a processor in a depth map generation system, where the depth map generation system includes: a sensor, a processor, and a plurality of projectors, the method comprising:
acquiring an image to be processed, wherein the image to be processed comprises speckles formed by reflecting light rays projected by a plurality of projectors;
comparing the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel of each speckle position in the image to be processed, wherein the preset speckle image comprises the initial reference position of each pixel of each speckle position;
aiming at any baseline, calculating the depth value of each pixel of each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel of each speckle position, wherein the baseline represents the distance between a projector and the sensor, and the distances between different projectors and the sensor are different;
aiming at any base line, generating a depth map corresponding to the base line according to the depth value of each pixel of each speckle position corresponding to the base line;
and fusing the depth maps corresponding to the baselines to obtain a target depth map.
Optionally, for any baseline, calculating a depth value of each pixel at each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel at each speckle position includes:
aiming at any baseline, calculating the depth value of each pixel of a plurality of speckle positions according to the baseline and the parallax corresponding to each pixel of each speckle position;
and aiming at any baseline, extracting the depth values of the speckles with the depth values within the preset depth range according to the preset depth range corresponding to the baseline to obtain the depth values of the pixels at the positions of the speckles corresponding to the baseline, wherein each baseline corresponds to one preset depth range, and the preset depth ranges of different baselines are different.
Optionally, for any baseline, calculating a depth value of each pixel at each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel at each speckle position includes:
aiming at any base line, selecting a plane equation corresponding to the base line;
and calculating the depth of each pixel of each speckle position according to the selected plane equation and the parallax corresponding to each pixel of each speckle position to obtain the depth value of each pixel of each speckle position.
In a third aspect of the embodiments of the present application, there is provided an autonomous mobile device, including a depth map generation system, where the depth map generation system includes: a sensor, a processor, and a plurality of projectors;
the projectors are used for projecting light rays;
the sensor is used for acquiring an image to be processed, wherein the image to be processed comprises speckles formed by reflecting light rays projected by the plurality of projectors;
the processor is used for comparing the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel of each speckle position in the image to be processed, wherein the preset speckle image comprises the initial reference position of each pixel of each speckle position; aiming at any baseline, calculating the depth value of each pixel of each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel of each speckle position, wherein the baseline represents the distance between a projector and the sensor, and the distances between different projectors and the sensor are different; aiming at any baseline, generating a depth map corresponding to the baseline according to the depth value of each pixel of each speckle position corresponding to the baseline; and fusing the depth maps corresponding to the baselines to obtain a target depth map.
Optionally, the processor is specifically configured to, for any baseline, calculate a depth value of each pixel of the plurality of speckle positions according to the baseline and a disparity corresponding to each pixel of each speckle position; and aiming at any baseline, extracting the depth values of the speckles with the depth values within the preset depth range according to the preset depth range corresponding to the baseline to obtain the depth values of the pixels at the positions of the speckles corresponding to the baseline, wherein each baseline corresponds to one preset depth range, and the preset depth ranges of different baselines are different.
Optionally, the processor is specifically configured to select, for any baseline, a plane equation corresponding to the baseline; and calculating the depth of each pixel of each speckle position according to the selected plane equation and the parallax corresponding to each pixel of each speckle position to obtain the depth value of each pixel of each speckle position.
Optionally, the plurality of projectors are specifically configured to alternately project light through each of the projectors;
the sensor is specifically used for acquiring the image to be processed once every preset time interval.
Optionally, an exposure time period of a projector closest to the sensor among the plurality of projectors is shorter than an exposure time period of a projector farthest from the sensor.
In another aspect of the embodiments of the present application, a computer-readable storage medium is provided, where a computer program is stored in the computer-readable storage medium, and when executed by a processor, the computer program implements any one of the depth map generating methods described above.
Embodiments of the present invention further provide a computer program product containing instructions, which when run on a computer, cause the computer to perform any one of the above-described depth map generation methods.
The embodiment of the invention has the following beneficial effects:
the embodiment of the invention provides a depth map generation system, a depth map generation method and autonomous mobile equipment, wherein the depth map generation system comprises a sensor, a processor and a plurality of projectors; the projectors are used for projecting light rays; the sensor is used for acquiring an image to be processed, wherein the image to be processed comprises speckles formed by reflecting light rays projected by the plurality of projectors; the processor is used for comparing the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel of each speckle position in the image to be processed, wherein the preset speckle image comprises the initial reference position of each pixel of each speckle position; aiming at any baseline, calculating the depth value of each pixel of each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel of each speckle position, wherein the baseline represents the distance between a projector and the sensor, and the distances between different projectors and the sensor are different; aiming at any baseline, generating a depth map corresponding to the baseline according to the depth value of each pixel of each speckle position corresponding to the baseline; and fusing the depth maps corresponding to the baselines to obtain a target depth map. Through the depth map generation system of the embodiment of the application, the corresponding depth maps can be generated respectively by utilizing a plurality of baselines with different distances between a plurality of projectors and the sensor, and the generated depth maps are fused, so that the accuracy of depth calculation is improved.
Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by referring to these drawings.
Fig. 1 is a schematic structural diagram of a depth map generation system according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of a baseline provided in an embodiment of the present application
FIG. 3 is a depth range mapping for different baselines provided by an embodiment of the present application;
fig. 4 is another schematic structural diagram of a depth map generating system according to an embodiment of the present disclosure;
fig. 5 is a schematic flowchart of a depth map generation method according to an embodiment of the present disclosure;
fig. 6 is another schematic flow chart of a depth map generation method according to an embodiment of the present disclosure;
fig. 7 is a schematic structural diagram of an autonomous mobile apparatus according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived from the embodiments given herein by one of ordinary skill in the art, are within the scope of the invention.
First, terms of art that may be used in the embodiments of the present application are explained:
baseline: the horizontal distance between the centers of the two camera apertures, or the center of the camera aperture and the center of the speckle projector, is referred to as the baseline of the depth camera.
Parallax: the binocular parallax is also called stereo parallax, and the closer an object is to an image sensor, the greater the difference between the two sensors collecting the object, which forms the binocular parallax. Subsequent algorithms may estimate the distance of the object to the camera using measurements of this disparity.
A speckle projector: apparatus for projecting speckle on a subject.
In a first aspect of the embodiments of the present application, there is provided a depth map generating system, see fig. 1, including a sensor 102, a processor 101, and a plurality of projectors 103;
a plurality of projectors 103 for projecting light;
the sensor 102 is configured to acquire an image to be processed, where the image to be processed includes speckles obtained by reflecting light rays projected by the plurality of projectors;
the processor 101 is configured to compare the image to be processed with a preset speckle image to obtain a parallax corresponding to each pixel at each speckle position in the image to be processed, where the preset speckle image includes an initial reference position of each pixel at each speckle position; aiming at any baseline, calculating the depth value of each pixel of each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel of each speckle position, wherein the baseline represents the distance between the projector and the sensor, and the distances between different projectors and sensors are different; aiming at any baseline, generating a depth map corresponding to the baseline according to the depth value of each pixel of each speckle position corresponding to the baseline; and fusing the depth maps corresponding to the baselines to obtain a target depth map.
The plurality of projectors in the embodiment of the present application may be physically independent projectors. In actual use, a plurality of projectors should be installed on the same horizontal line as much as possible, so that the time loss in calculation is reduced. When the projector is mounted on the smart mobile device, the plurality of projectors and sensors may be mounted on the same side or different sides of the smart mobile device. In which the distance between each projector and the Sensor is different in the plurality of projectors, and thus the formed base lines are also different, for example, referring to fig. 2, the distances between the projector 1, the projector 2, the projector 3, and the Sensor (Sensor) are s1, s2, s3, respectively, three base lines s1, s2, s3 can be formed.
After the to-be-processed image is acquired by the sensor, the acquired to-be-processed image can be preprocessed, and specifically, image features and brightness equalization can be added in the modes of contrast enhancement, histogram equalization, binarization and the like.
The image to be processed is compared with the preset speckle image to obtain the parallax corresponding to each pixel at each speckle position in the image to be processed, and the parallax corresponding to each pixel at each speckle position in the image to be processed collected by the sensor is compared with the preset speckle image to obtain the parallax corresponding to each pixel at each speckle position and generate the corresponding parallax image. When the depth of each speckle is calculated according to any baseline and each pixel of each speckle position to obtain the depth value of each speckle, the corresponding depth value can be calculated according to the generated parallax image and a preset plane equation to generate a depth image. For example, image matching calculation is performed on a speckle image acquired by a sensor and a reference plane speckle image recorded in a calibration stage to obtain a disparity map; and calculating a depth map by using the parallax map, the reference plane equation and the camera focal length and the distance between the speckles and the projector.
For any baseline, the depth value of each pixel at each speckle position corresponding to the baseline is calculated according to the baseline and the parallax corresponding to each pixel at each speckle position, and the depth value can be calculated through a plurality of preset algorithms, for example, through a monocular depth algorithm. For example, depth calculations are performed two by two between the Sensor and the projector; the Sensor and the projector 1 calculate a depth map m1 by using a monocular depth calculation method and taking S1 as a baseline; the Sensor and the projector 2 calculate a depth map m2 by using a monocular depth calculation method and taking S2 as a baseline; the Sensor and projector 3 calculate the depth map m3 using monocular depth calculation using S3 as a baseline. Specifically, the calculation process can be referred to in the following embodiments.
In the embodiment of the present application, the inventor finds that, when the depth values are calculated according to different base lines, the optimal precision corresponding to the different base lines is often a certain depth range, see fig. 3. Therefore, when the depth map is calculated and generated based on the baselines, only the depth within a certain distance range is calculated for each baseline, thereby improving the calculation accuracy. And then, fusing the depth maps generated by the baselines to obtain a target depth map, so that a complete depth map is obtained through fusion.
Compared with the prior art, in order to improve the accuracy of depth calculation, through setting up a plurality of sensors, not only lead to need to insert a large amount of image sensor this to SOC (System on Chip)'s requirement very high, use a plurality of SOCs moreover again and relate to the synchronous problem System complexity height of many SOCs, be unfavorable for the System be real-time and problematic nature, simultaneously also have the cost increase problem. By the method of the embodiment of the application, the problem of improving the accuracy of depth calculation can be well solved through the multi-projector, and the projector generally only needs PWM (Pulse width modulation) control, so that the circuit is simple and easy to control.
Therefore, through the depth map generation system provided by the embodiment of the application, the corresponding depth maps can be respectively calculated by utilizing a plurality of baselines with different distances between a plurality of projectors and a sensor, and the calculated depth maps are fused, so that the accuracy of depth calculation is improved.
Optionally, the processor is specifically configured to select, for any baseline, a plane equation corresponding to the baseline; and calculating the depth of each pixel of each speckle position according to the selected plane equation and the parallax corresponding to each pixel of each speckle position to obtain the depth value of each pixel of each speckle position.
In the actual use process, the sensor internal reference correction image can be calibrated in advance. For example, a reference plane is selected, and the reference is determined by existing camera calibration algorithmsThe plane equation of the plane in the camera coordinate system, and the baseline distances s1, s2, s3 of the projector and the sensor. The projectors project the speckle images to the reference plane, the reference images R1, R2, R3 are recorded for the respective projectors, respectively, according to the maximum parallax calculation value dmax set by the algorithm, the depth intervals [0, d1 ], [ d1, d2 ], [ d2, d3 ], [ d3, + ∞),
and f is the equivalent focal length after the camera is corrected, so that the calibration of the sensor and the projector is realized pairwise.
When the depth of each speckle is calculated through a plane equation and the parallax, the depth corresponding to different parallaxes needs to be acquired and calculated according to the corresponding relation among the base line, the parallax, the view field, the depth and the like. Therefore, in the actual use process, a plane equation can be created in advance for different baselines, and the plane equation can represent the corresponding relationship between the parallax and the depth corresponding to the baselines. Therefore, when the depth is calculated, the corresponding plane equation can be selected according to the base line, and then the corresponding depth is calculated according to the parallax, so that the calculation efficiency and the calculation precision are improved.
Optionally, the processor is specifically configured to, for any baseline, calculate a depth value of each pixel of the plurality of speckle positions according to the baseline and a disparity corresponding to each pixel of each speckle position; and aiming at any baseline, extracting the depth values of the speckles with the depth values within the preset depth range according to the preset depth range corresponding to the baseline to obtain the depth values of the pixels at the positions of the speckles corresponding to the baseline, wherein each baseline corresponds to one preset depth range, and the preset depth ranges of different baselines are different.
For example, depth calculations are performed two by two between the Sensor and the projector: the Sensor and the projector 1 calculate a depth map m1 by using a monocular depth calculation method and taking S1 as a baseline; the Sensor and the projector 2 calculate a depth map m2 by using a monocular depth calculation method and taking S2 as a baseline; the Sensor and projector 3 calculate the depth map m3 using monocular depth calculation with S3 as a baseline. In the fusion, m1 can be calculated by taking the minimum baseline s1 as a reference, and the final depth map m determines which depth map to select the corresponding value from according to the depth intervals [0, d1 ], [ d1, d2 ], [ d2, d3 ], [ d3, + ∞) determined when each point in m1 falls on the calibration. The formula is as follows:
finally, for any baseline, a depth map is generated according to the extracted brightness values.
Optionally, a plurality of projectors, in particular for projecting light alternately through the projectors;
the sensor is specifically used for acquiring the image to be processed once every preset time interval.
Optionally, in the plurality of projectors, an exposure time period of a projector closest to the sensor is shorter than an exposure time period of a projector farthest from the sensor.
In actual use, the exposure time periods of the different projectors may be proportional to the corresponding baselines. For example, taking 3 projectors as an example, the corresponding baselines are d1, d2, and d3, respectively, the exposure times t1, t2, and t3 corresponding to the three projectors can be calculated by the following steps:
if the time of one frame of the synthesized depth map is T, T1+ T2+ T3 is T;
t1/t2/t3＝d1/d2/d3；
then, T1 ═ d1 × T/(d1+ d2+ d 3);
t2＝d2*T/(d1+d2+d3)；
t3＝d3*T/(d1+d2+d3)。
referring to fig. 4, the depth map generating system includes an image sensor, a plurality of projectors, an image signal processing module, a depth image processing module, a digital signal processing module, and a projector control module.
And the projector control module is used for controlling the plurality of projectors to alternately encode the space. Each projector may correspond to an image signal processor, the image encoded by the projector 1 is input to the image signal processing module 1, the image encoded by the projector 2 is input to the image signal processing module 2, and the image encoded by the projector 3 is input to the image signal processing module 3. The image signal processing module 1, the image signal processing module 2 and the image signal processing module 3 can be 3 physically independent modules or the same module for time division multiplexing. Similarly, the DPUs (depth image processing modules) 1, 2, 3, 1, and 2 may be physically independent modules or may be time-division multiplexed by the same module.
For the 3 projector image acquisition process, taking an output frame rate of 30FPS as an example, the total duration for acquiring 3 projector image frames is 1/30 seconds, and the acquisition time for averaging one frame is 1/90 seconds. The exposure duration of the short baseline projector image frames is less than the exposure duration of the long baseline projector image frames, considering that the projection energy of the projector is smaller with distance, for more uniform distance and higher quality depth maps. For example, the projector image frame 1 exposure time period 5ms, the projector image frame 2 exposure time period 10ms, and the projector image frame 3 exposure time period 15 ms.
The multi-eye system can illuminate an object through an infrared structural light by the projector under the condition of lacking textures, artificial textures are projected on the surface of the object, a dual-mode depth acquisition mode is utilized to acquire a depth map, the quality is higher, the reliability is higher, the distances s1 and s2 between the projector and the cameras are smaller than the distance s3 between the two cameras under the common condition, the structural light intensity at a short distance can be acquired by a monocular mode under the outdoor high-brightness environment, and the quality of the depth map after fusion is acquired by structural light attenuation at the long distance by a binocular depth calculation mode directly is higher than that of the traditional structural light or a binocular scheme.
For the purpose of illustrating the aspects of the present application, reference is made to fig. 5, which is described below in connection with specific examples:
the depth map generation method comprises the following steps:
firstly, off-line processing:
off-line demarcating the sensor and the projector in pairs: a reference plane is selected, and a plane equation F of the reference plane in the camera coordinate system and baseline distances s1, s2, s3 of the projector and the sensor are determined by existing camera calibration algorithms.
The projectors project the speckle images onto a reference plane, the reference images R1, R2, R3 being recorded for each projector, respectively; calculating a depth interval [0, d1 ], [ d1, d2 ], [ d2, d3 ], [ d3, + ∞) according to the maximum parallax calculation value dmax set by the algorithm,
wherein f is the equivalent focal length after the camera is corrected.
Secondly, online processing:
the method comprises the following steps: the laser speckle projector performs spatial encoding.
Step two: the sensor collects and preprocesses images, and the preprocessing method comprises the steps of increasing image characteristics and conducting brightness equalization in the modes of contrast enhancement, histogram equalization, binaryzation and the like.
Step three: depth calculation is carried out between the sensor and the projector in pairs: the Sensor and the projector 1 calculate the depth map m1 using the monocular depth computing method with S1 as a baseline, the Sensor and the projector 2 calculate the depth map m2 using the monocular depth computing method with S2 as a baseline, and the Sensor and the projector 3 calculate the depth map m3 using the monocular depth computing method with S3 as a baseline.
The monocular depth is calculated as follows: utilizing the camera internal reference calibrated off-line to correct the image; performing image matching calculation on the speckle images acquired by the camera and the reference plane speckle images recorded in the calibration stage to obtain a disparity map; and calculating a depth map by using the parallax map, the reference plane equation and the camera focal length and the distance between the speckles and the projector.
Step four: performing fusion operation on the 3 depth maps obtained in the previous step: with reference to m1 calculated from the minimum baseline s1, the final depth map m determines from which depth map the corresponding value is selected according to the depth interval [0, d1 ], [ d1, d2 ], [ d2, d3 ], [ d3, + ∞) range determined when each point in m1 falls within the calibration.
In a second aspect of the embodiments of the present application, a depth map generation method is provided, which is applied to a processor in a depth map generation system, where the depth map generation system includes: a sensor, a processor and a plurality of projectors, see fig. 6, the method comprising:
step S61, acquiring an image to be processed, wherein the image to be processed comprises speckles obtained after the light rays projected by a plurality of projectors are reflected;
step S62, comparing the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel of each speckle position in the image to be processed, wherein the preset speckle image comprises the initial reference position of each pixel of each speckle position;
step S63, aiming at any base line, calculating the depth value of each pixel of each speckle position corresponding to the base line according to the base line and the parallax corresponding to each pixel of each speckle position, wherein the base line represents the distance between the projector and the sensor, and the distances between different projectors and sensors are different;
step S64, aiming at any base line, generating a depth map corresponding to the base line according to the depth value of each pixel of each speckle position corresponding to the base line;
and step S65, fusing the depth maps corresponding to the base lines to obtain a target depth map.
Optionally, for any baseline, calculating a depth value of each pixel at each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel at each speckle position, including:
aiming at any baseline, calculating the depth value of each pixel of a plurality of speckle positions according to the baseline and the parallax corresponding to each pixel of each speckle position;
and aiming at any baseline, extracting the depth values of the speckles with the depth values within the preset depth range according to the preset depth range corresponding to the baseline to obtain the depth values of the pixels at the positions of the speckles corresponding to the baseline, wherein each baseline corresponds to one preset depth range, and the preset depth ranges of different baselines are different.
Optionally, for any baseline, calculating a depth value of each pixel at each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel at each speckle position, including:
aiming at any base line, selecting a plane equation corresponding to the base line;
and calculating the depth of each pixel of each speckle position according to the selected plane equation and the parallax corresponding to each pixel of each speckle position to obtain the depth value of each pixel of each speckle position.
Therefore, by the depth map generation method, the corresponding depth maps can be generated by using the plurality of baselines with different distances between the plurality of projectors and the sensors respectively, and the generated depth maps are fused, so that the accuracy of depth calculation is improved.
In a third aspect of the embodiments of the present application, there is provided an autonomous mobile device, including a depth map generation system, where the depth map generation system includes: a sensor, a processor, and a plurality of projectors;
a plurality of projectors for projecting light;
the sensor is used for acquiring an image to be processed, wherein the image to be processed comprises speckles formed by reflecting light rays projected by the plurality of projectors;
the processor is used for comparing the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel of each speckle position in the image to be processed, wherein the preset speckle image comprises the initial reference position of each pixel of each speckle position; aiming at any baseline, calculating the depth value of each pixel of each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel of each speckle position, wherein the baseline represents the distance between the projector and the sensor, and the distances between different projectors and sensors are different; aiming at any baseline, generating a depth map corresponding to the baseline according to the depth value of each pixel of each speckle position corresponding to the baseline; and fusing the depth maps corresponding to the baselines to obtain a target depth map.
Optionally, the processor is specifically configured to, for any baseline, calculate a depth value of each pixel of the plurality of speckle positions according to the baseline and a parallax corresponding to each pixel of each speckle position; and aiming at any baseline, extracting the depth values of the speckles with the depth values within the preset depth range according to the preset depth range corresponding to the baseline to obtain the depth values of the pixels at the positions of the speckles corresponding to the baseline, wherein each baseline corresponds to one preset depth range, and the preset depth ranges of different baselines are different.
Optionally, the processor is specifically configured to select, for any baseline, a plane equation corresponding to the baseline; and calculating the depth of each pixel of each speckle position according to the selected plane equation and the parallax corresponding to each pixel of each speckle position to obtain the depth value of each pixel of each speckle position.
Optionally, a plurality of projectors, in particular for projecting light alternately through the projectors;
the sensor is specifically used for acquiring the image to be processed once every preset time interval.
Optionally, the exposure time period of the projector closest to the sensor among the plurality of projectors is shorter than the exposure time period of the projector farthest from the sensor.
Specifically, the autonomous moving device may be a mobile robot, a sweeping robot, or other devices that can move automatically. The automatic mobile device can measure the depth and generate the depth map according to the depth map generating system, and then autonomously move or stop according to the detection result, for example, referring to fig. 7, the front and rear ends of the automatic mobile device can be respectively provided with a plurality of projectors and a detector, the front and rear projectors can be controlled by a processor to project speckles, the front and rear sensors can collect speckle images, the collected images are input into the processor to be calculated, and then corresponding depth calculation results, such as clustering of obstacles in front of or behind, can be obtained, so that the automatic mobile device stops when an obstacle is found in front of the automatic mobile device through depth detection, or continues to advance after the obstacle in front is found to be removed through depth detection.
Therefore, through the processor of the embodiment of the application, the corresponding depth maps can be respectively generated by utilizing the multiple baselines with different distances between the multiple projectors and the sensors, and the generated depth maps are fused, so that the accuracy of depth calculation is improved.
An embodiment of the present invention further provides an electronic device, as shown in fig. 8, which includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804, where the processor 801, the communication interface 802, and the memory 803 complete mutual communication through the communication bus 804,
a memory 803 for storing a computer program;
the processor 801 is configured to implement the following steps when executing the program stored in the memory 803:
acquiring an image to be processed, wherein the image to be processed comprises speckles formed by reflecting light rays projected by a plurality of projectors;
comparing the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel of each speckle position in the image to be processed, wherein the preset speckle image comprises the initial reference position of each pixel of each speckle position;
aiming at any baseline, calculating the depth value of each pixel of each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel of each speckle position, wherein the baseline represents the distance between the projector and the sensor, and the distances between different projectors and sensors are different;
aiming at any baseline, generating a depth map corresponding to the baseline according to the depth value of each pixel of each speckle position corresponding to the baseline;
and fusing the depth maps corresponding to the baselines to obtain a target depth map.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
In a further embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program, when executed by a processor, implements the steps of any one of the above-mentioned depth map generation methods.
In a further embodiment provided by the present invention, there is also provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform any of the depth map generation methods of the embodiments described above.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It should be noted that, in this document, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the method, autonomous mobile device, computer-readable storage medium, and computer program product embodiments are described with relative simplicity as they are substantially similar to the method embodiments, with respect to which reference is made to some of the descriptions of the method embodiments.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (14)
1. A depth map generation system comprising a sensor, a processor and a plurality of projectors;
the projectors are used for projecting light rays;
the sensor is used for acquiring an image to be processed, wherein the image to be processed comprises speckles formed by reflecting light rays projected by the plurality of projectors;
the processor is used for comparing the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel of each speckle position in the image to be processed, wherein the preset speckle image comprises the initial reference position of each pixel of each speckle position; aiming at any baseline, calculating the depth value of each pixel of each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel of each speckle position, wherein the baseline represents the distance between a projector and the sensor, and the distances between different projectors and the sensor are different; aiming at any baseline, generating a depth map corresponding to the baseline according to the depth value of each pixel of each speckle position corresponding to the baseline; and fusing the depth maps corresponding to the baselines to obtain a target depth map.
2. The system of claim 1,
the processor is specifically configured to calculate, for any baseline, a depth value of each pixel of the plurality of speckle positions according to the baseline and a disparity corresponding to each pixel of each speckle position; and aiming at any baseline, extracting the depth values of the speckles with the depth values within the preset depth range according to the preset depth range corresponding to the baseline to obtain the depth values of the pixels at the positions of the speckles corresponding to the baseline, wherein each baseline corresponds to one preset depth range, and the preset depth ranges of different baselines are different.
3. The system of claim 1,
the processor is specifically configured to select, for any baseline, a plane equation corresponding to the baseline; and calculating the depth of each pixel of each speckle position according to the selected plane equation and the parallax corresponding to each pixel of each speckle position to obtain the depth value of each pixel of each speckle position.
4. The system of claim 1,
the projectors are used for alternately projecting light rays through the projectors;
the sensor is specifically used for acquiring the image to be processed once every preset time interval.
5. The system of claim 1,
the exposure time period of a projector closest to the sensor among the plurality of projectors is shorter than the exposure time period of a projector farthest from the sensor.
6. A depth map generation method applied to a processor in a depth map generation system, the depth map generation system comprising: a sensor, a processor, and a plurality of projectors, the method comprising:
acquiring an image to be processed, wherein the image to be processed comprises speckles formed by reflecting light rays projected by a plurality of projectors;
comparing the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel of each speckle position in the image to be processed, wherein the preset speckle image comprises the initial reference position of each pixel of each speckle position;
aiming at any baseline, calculating the depth value of each pixel of each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel of each speckle position, wherein the baseline represents the distance between a projector and the sensor, and the distances between different projectors and the sensor are different;
aiming at any baseline, generating a depth map corresponding to the baseline according to the depth value of each pixel of each speckle position corresponding to the baseline;
and fusing the depth maps corresponding to the baselines to obtain a target depth map.
7. The method according to claim 6, wherein for any baseline, calculating a depth value of each pixel at each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel at each speckle position comprises:
aiming at any baseline, calculating the depth value of each pixel of a plurality of speckle positions according to the baseline and the parallax corresponding to each pixel of each speckle position;
and aiming at any baseline, extracting the depth values of the speckles with the depth values within the preset depth range according to the preset depth range corresponding to the baseline to obtain the depth values of the pixels at the positions of the speckles corresponding to the baseline, wherein each baseline corresponds to one preset depth range, and the preset depth ranges of different baselines are different.
8. The method according to claim 6, wherein for any baseline, calculating a depth value of each pixel at each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel at each speckle position comprises:
aiming at any base line, selecting a plane equation corresponding to the base line;
and calculating the depth of each pixel of each speckle position according to the selected plane equation and the parallax corresponding to each pixel of each speckle position to obtain the depth value of each pixel of each speckle position.
9. An autonomous mobile device comprising a depth map generation system, the depth map generation system comprising: a sensor, a processor, and a plurality of projectors;
the projectors are used for projecting light rays;
the sensor is used for acquiring an image to be processed, wherein the image to be processed comprises speckles formed by reflecting light rays projected by the plurality of projectors;
the processor is used for comparing the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel of each speckle position in the image to be processed, wherein the preset speckle image comprises the initial reference position of each pixel of each speckle position; aiming at any baseline, calculating the depth value of each pixel of each speckle position corresponding to the baseline according to the baseline and the parallax corresponding to each pixel of each speckle position, wherein the baseline represents the distance between a projector and the sensor, and the distances between different projectors and the sensor are different; aiming at any baseline, generating a depth map corresponding to the baseline according to the depth value of each pixel of each speckle position corresponding to the baseline; and fusing the depth maps corresponding to the baselines to obtain a target depth map.
10. The autonomous mobile device of claim 9,
the processor is specifically configured to calculate, for any baseline, a depth value of each pixel of the plurality of speckle positions according to the baseline and a parallax corresponding to each pixel of each speckle position; and aiming at any baseline, extracting the depth values of the speckles with the depth values within the preset depth range according to the preset depth range corresponding to the baseline to obtain the depth values of the pixels at the positions of the speckles corresponding to the baseline, wherein each baseline corresponds to one preset depth range, and the preset depth ranges of different baselines are different.
11. The autonomous mobile device of claim 9,
the processor is specifically configured to select, for any baseline, a plane equation corresponding to the baseline; and calculating the depth of each pixel of each speckle position according to the selected plane equation and the parallax corresponding to each pixel of each speckle position to obtain the depth value of each pixel of each speckle position.
12. The autonomous mobile device of claim 9,
the projectors are used for alternately projecting light rays through the projectors;
the sensor is specifically used for acquiring the image to be processed once every preset time interval.
13. The autonomous mobile device of claim 9,
the exposure time period of a projector of the plurality of projectors closest to the sensor is less than the exposure time period of a projector furthest from the sensor.
14. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of the claims 6-8.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210328464.XA CN114627174A (en) | 2022-03-30 | 2022-03-30 | Depth map generation system and method and autonomous mobile device |
| PCT/CN2023/079554 WO2023185375A1 (en) | 2022-03-30 | 2023-03-03 | Depth map generation system and method, and autonomous mobile device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210328464.XA CN114627174A (en) | 2022-03-30 | 2022-03-30 | Depth map generation system and method and autonomous mobile device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114627174A true CN114627174A (en) | 2022-06-14 |
Family
ID=81904659
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210328464.XA Pending CN114627174A (en) | 2022-03-30 | 2022-03-30 | Depth map generation system and method and autonomous mobile device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN114627174A (en) |
| WO (1) | WO2023185375A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023185375A1 (en) * | 2022-03-30 | 2023-10-05 | 杭州萤石软件有限公司 | Depth map generation system and method, and autonomous mobile device |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111726515A (en) * | 2019-03-22 | 2020-09-29 | 爱唯秀股份有限公司 | Depth camera system |
| CN112150528A (en) * | 2019-06-27 | 2020-12-29 | Oppo广东移动通信有限公司 | Depth image acquisition method, terminal and computer readable storage medium |
| CN112927280B (en) * | 2021-03-11 | 2022-02-11 | 北京的卢深视科技有限公司 | Method and device for acquiring depth image and monocular speckle structured light system |
| CN113311451B (en) * | 2021-05-07 | 2024-01-16 | 西安交通大学 | Laser speckle projection TOF depth perception method and device |
| CN113936050B (en) * | 2021-10-21 | 2022-08-12 | 合肥的卢深视科技有限公司 | Speckle image generation method, electronic device, and storage medium |
| CN114627174A (en) * | 2022-03-30 | 2022-06-14 | 杭州萤石软件有限公司 | Depth map generation system and method and autonomous mobile device |
-
2022
- 2022-03-30 CN CN202210328464.XA patent/CN114627174A/en active Pending
-
2023
- 2023-03-03 WO PCT/CN2023/079554 patent/WO2023185375A1/en unknown
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023185375A1 (en) * | 2022-03-30 | 2023-10-05 | 杭州萤石软件有限公司 | Depth map generation system and method, and autonomous mobile device |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2023185375A1 (en) | 2023-10-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3416370B1 (en) | Photography focusing method, device, and apparatus for terminal | |
| US8718326B2 (en) | System and method for extracting three-dimensional coordinates | |
| US9424650B2 (en) | Sensor fusion for depth estimation | |
| CN105627932B (en) | A kind of distance measuring method and device based on binocular vision | |
| KR100776649B1 (en) | A depth information-based Stereo/Multi-view Stereo Image Matching Apparatus and Method | |
| US8144974B2 (en) | Image processing apparatus, method, and program | |
| WO2021063128A1 (en) | Method for determining pose of active rigid body in single-camera environment, and related apparatus | |
| CN105578019A (en) | Image extraction system capable of obtaining depth information and focusing method | |
| CN108924408B (en) | Depth imaging method and system | |
| TWI529661B (en) | Method of quickly building up depth map and image processing device | |
| CN109840922B (en) | Depth acquisition method and system based on binocular light field camera | |
| CN110213491B (en) | Focusing method, device and storage medium | |
| CN109636857B (en) | Alignment method and calibration system | |
| CN105472237A (en) | Imaging apparatus and imaging method | |
| CN111798507A (en) | Power transmission line safety distance measuring method, computer equipment and storage medium | |
| WO2023185375A1 (en) | Depth map generation system and method, and autonomous mobile device | |
| US11879993B2 (en) | Time of flight sensor module, method, apparatus and computer program for determining distance information based on time of flight sensor data | |
| CN113014899B (en) | Binocular image parallax determination method, device and system | |
| US11295421B2 (en) | Image processing method, image processing device and electronic device | |
| US11283970B2 (en) | Image processing method, image processing apparatus, electronic device, and computer readable storage medium | |
| KR20180053718A (en) | Apparatus and method for generating data representing a pixel beam | |
| CN112750157A (en) | Depth image generation method and device | |
| Fleck et al. | Graph cut based panoramic 3D modeling and ground truth comparison with a mobile platform–The Wägele | |
| US11790600B2 (en) | Image processing device, imaging apparatus, image processing method, and recording medium | |
| CN114845043B (en) | Automatic focusing method, system, electronic device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |