WO2023185375A1

WO2023185375A1 - Depth map generation system and method, and autonomous mobile device

Info

Publication number: WO2023185375A1
Application number: PCT/CN2023/079554
Authority: WO
Inventors: 杜斌
Original assignee: 杭州萤石软件有限公司
Priority date: 2022-03-30
Filing date: 2023-03-03
Publication date: 2023-10-05
Also published as: CN114627174A

Abstract

A depth map generation system and method, and an autonomous mobile device. The method comprises: a plurality of projectors (103) projecting light; a sensor (102) acquiring an image to be processed; a processor (101) comparing the image to be processed with a preset speckle image, so as to obtain disparities corresponding to pixels at speckle locations in the image to be processed; for any baseline, on the basis of the baseline and the disparities corresponding to the pixels at the speckle locations, calculating depth values, which correspond to the baseline, of the pixels at the speckle locations; for any baseline, on the basis of the depth values, which correspond to the baseline, of the pixels at the speckle locations, generating a depth map corresponding to the baseline; and fusing depth maps corresponding to baselines, so as to obtain a target depth map. Therefore, a plurality of baselines for different distances between a plurality of projectors (103) and a sensor (102) can be used to respectively calculate corresponding depth maps, and the calculated depth maps are fused, thereby improving the accuracy of depth calculation.

Description

A depth map generation system, method and autonomous mobile device

This application claims the priority of the Chinese patent application with the application number 202210328464. This reference is incorporated into this application.

Technical field

This application relates to the field of information technology, and in particular to a depth map generation system, method and autonomous mobile device.

Background technique

At present, with the popularization of smart devices, the application of automatic obstacle avoidance equipment has become more and more widespread. For example, a sweeping robot can recognize and avoid obstacles around it. When identifying obstacles, it is necessary to project speckles through a projector, then collect images including speckles through sensors, and finally use the collected images to perform depth calculations and generate depth maps, through which obstacles can be identified and avoided.

In the existing technology, in order to improve the accuracy of depth calculation, it is generally necessary to arrange multiple sensors for collection, use baselines between different sensors and projectors, and use a three-dimensional distance measurement method to perform measurements. However, this method not only places high requirements on the chip, but also leads to high system complexity.

Contents of the invention

The purpose of the embodiments of the present application is to provide a depth map generation system, method and autonomous mobile device to improve the accuracy of depth calculation. The specific technical solutions are as follows:

The first aspect of the embodiment of the present application first provides a depth map generation system, including a sensor, a processor and multiple projectors;

The plurality of projectors are used to project light;

The sensor is used to collect an image to be processed, wherein the image to be processed includes speckles after reflection of light projected by multiple projectors;

The processor is configured to compare the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel at each speckle position in the image to be processed, wherein the preset speckle image Including the initial reference position of each pixel at each speckle position; for any baseline, calculate the depth value of each pixel at each speckle position corresponding to the baseline based on the disparity corresponding to the baseline and each pixel at each speckle position. , where the baseline represents the distance between the projector and the sensor, different projection The distance between the detector and the sensor is different; for any baseline, a depth map corresponding to the baseline is generated according to the depth value of each pixel at each speckle position corresponding to the baseline; and the depth map corresponding to each baseline is processed. Fusion to obtain the target depth map.

Optionally, the processor is specifically configured to calculate, for any baseline, the depth value of each pixel at multiple speckle locations based on the baseline and the parallax corresponding to each pixel at each speckle location; for any Baseline, according to the preset depth range corresponding to the baseline, extract the depth value of the speckle whose depth value is within the preset depth range, and obtain the depth value of each pixel at each speckle position corresponding to the baseline, where each baseline Corresponds to a preset depth range, and different baselines correspond to different preset depth ranges.

Optionally, the processor is specifically configured to select a plane equation corresponding to any baseline; and calculate the parallax of each speckle position according to the selected plane equation and the parallax corresponding to each pixel of each speckle position. The depth of each pixel is used to obtain the depth value of each pixel at each speckle position.

Optionally, the plurality of projectors are specifically used to alternately project light through each of the projectors;

The sensor is specifically used to collect images to be processed every preset time interval.

Optionally, the exposure time of the projector closest to the sensor among the plurality of projectors is shorter than the exposure time of the projector farthest from the sensor.

A second aspect of the embodiment of the present application provides a depth map generation method, which is applied to a processor in a depth map generation system. The depth map generation system includes: a sensor, a processor, and multiple projectors. The method include:

Obtaining an image to be processed, wherein the image to be processed includes reflected speckles of light projected by multiple projectors;

The image to be processed is compared with a preset speckle image to obtain the parallax corresponding to each pixel at each speckle position in the image to be processed, wherein the preset speckle image includes each pixel at each speckle position. The initial reference position of the pixel;

For any baseline, calculate the depth value of each pixel at each speckle position corresponding to the baseline based on the disparity corresponding to the baseline and each pixel at each speckle position, where the baseline represents the projector and the sensor The distance between different projectors and the sensor is different;

For any baseline, generate a depth map corresponding to the baseline based on the depth value of each pixel at each speckle position corresponding to the baseline;

The depth maps corresponding to each of the baselines are fused to obtain a target depth map.

Optionally, for any baseline, each pixel pair according to the baseline and each speckle position According to the corresponding disparity, calculate the depth value of each pixel at each speckle position corresponding to the baseline, including:

For any baseline, calculate the depth values of each pixel at multiple speckle locations based on the disparity corresponding to the baseline and each pixel at each speckle location;

For any baseline, according to the preset depth range corresponding to the baseline, extract the depth value of the speckle whose depth value is within the preset depth range, and obtain the depth value of each pixel at each speckle position corresponding to the baseline, where, Each baseline corresponds to a preset depth range, and different baselines correspond to different preset depth ranges.

Optionally, for any baseline, calculate the depth value of each pixel at each speckle position corresponding to the baseline based on the disparity corresponding to the baseline and each pixel at each speckle position, including:

For any baseline, select the plane equation corresponding to the baseline;

According to the selected plane equation and the parallax corresponding to each pixel at each speckle position, the depth of each pixel at each speckle position is calculated to obtain the depth value of each pixel at each speckle position.

A third aspect of the embodiments of the present application provides an autonomous mobile device, including a depth map generation system, where the depth map generation system includes: a sensor, a processor, and a plurality of projectors;

The plurality of projectors are used to project light;

The processor is configured to compare the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel at each speckle position in the image to be processed, wherein the preset speckle image Including the initial reference position of each pixel at each speckle position; for any baseline, calculate the depth value of each pixel at each speckle position corresponding to the baseline based on the disparity corresponding to the baseline and each pixel at each speckle position. , wherein the baseline represents the distance between the projector and the sensor, and the distance between different projectors and the sensor is different; for any baseline, the depth of each pixel at each speckle position corresponding to the baseline value, generate a depth map corresponding to the baseline; fuse the depth maps corresponding to each baseline to obtain a target depth map.

Another aspect of the embodiments of the present application provides a computer-readable storage medium, which is characterized in that a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned tasks are implemented. Depth map generation method described above.

Embodiments of the present application also provide a computer program product containing instructions that, when run on a computer, cause the computer to execute any one of the depth map generation methods described above.

Beneficial effects of the embodiments of this application:

Embodiments of the present application provide a depth map generation system, method and autonomous mobile device. The depth map generation system includes a sensor, a processor and multiple projectors; the multiple projectors are used to project light; the sensor , used to collect images to be processed, wherein the images to be processed include speckles after reflection of light projected by multiple projectors; the processor is used to compare the images to be processed with a preset speckle image By comparison, the disparity corresponding to each pixel at each speckle position in the image to be processed is obtained, wherein the preset speckle image includes the initial reference position of each pixel at each speckle position; for any baseline, according to the The parallax corresponding to the baseline and each pixel at each speckle position is calculated, and the depth value of each pixel at each speckle position corresponding to the baseline is calculated, where the baseline represents the distance between the projector and the sensor. Different projections The distance between the detector and the sensor is different; for any baseline, a depth map corresponding to the baseline is generated according to the depth value of each pixel at each speckle position corresponding to the baseline; and the depth map corresponding to each baseline is processed. Fusion to obtain the target depth map. Through the depth map generation system of the embodiment of the present application, multiple baselines with different distances between projectors and sensors can be used to generate corresponding depth maps respectively, and the generated depth maps can be fused, thereby improving the efficiency of depth calculation. Accuracy.

Of course, implementing any product or method of the present application does not necessarily require achieving all the above-mentioned advantages simultaneously.

Description of drawings

The drawings described here are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application.

Figure 1 is a schematic structural diagram of a depth map generation system provided by an embodiment of the present application;

Figure 2 is a schematic diagram of the baseline provided by the embodiment of the present application.

Figure 3 shows the corresponding relationship between the depth ranges of different baselines provided by the embodiment of the present application;

Figure 4 is another structural schematic diagram of a depth map generation system provided by an embodiment of the present application;

Figure 5 is a schematic flowchart of a depth map generation method provided by an embodiment of the present application;

Figure 6 is another schematic flowchart of a depth map generation method provided by an embodiment of the present application;

Figure 7 is a schematic structural diagram of an autonomous mobile device provided by an embodiment of the present application;

FIG. 8 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and examples. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art based on this application fall within the scope of protection of this application.

First, the professional terms that may be used in the embodiments of this application are explained:

Baseline: The horizontal distance between the centers of two camera apertures or the center of a camera aperture and the center of a speckle projector is called the baseline of a depth camera.

Parallax: Binocular parallax is also called stereoscopic parallax. The closer the object is to the image sensor (sensor), the greater the difference in the object collected by the two sensors, which forms binocular parallax. Subsequent algorithms can use measurements of this parallax to estimate the object's distance from the camera.

Speckle Projector: A device used to project speckles on a subject.

The first aspect of the embodiment of this application first provides a depth map generation system, see Figure 1. Includes a sensor 102, a processor 101 and a plurality of projectors 103;

A plurality of projectors 103 for projecting light;

The sensor 102 is used to collect images to be processed, where the images to be processed include speckles after reflection of light projected by multiple projectors;

The processor 101 is configured to compare the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel at each speckle position in the image to be processed, where the preset speckle image includes each pixel at each speckle position. The initial reference position of the pixel; for any baseline, the depth value of each pixel at each speckle position corresponding to the baseline is calculated based on the disparity corresponding to the baseline and each pixel at each speckle position, where the baseline represents the projector and sensor The distance between different projectors and sensors is different; for any baseline, a depth map corresponding to the baseline is generated based on the depth value of each pixel at each speckle position corresponding to the baseline; for each baseline The depth maps are fused to obtain the target depth map.

Among them, the multiple projectors in the embodiment of the present application may be physically independent multiple projectors. In actual use, multiple projectors should be installed on the same horizontal line as much as possible to reduce the time loss during calculation. When the projector is installed on the smart mobile device, multiple projectors and sensors can be installed on the same side or different sides of the smart mobile device. Among the multiple projectors, the distance between each projector and the sensor is different, so the baselines formed are also different. For example, see Figure 2, the distances between projector 1, projector 2, projector 3 and the sensor (Sensor) are respectively are s1, s2, and s3, then three baselines s1, s2, and s3 can be formed.

Among them, after the sensor collects the image to be processed, it can also preprocess the collected image to be processed. Specifically, the preprocessing methods include contrast enhancement, histogram equalization, binarization and other methods. Preprocessing can increase the Image features and brightness equalization.

Among them, the image to be processed is compared with the preset speckle image to obtain the parallax corresponding to each pixel of each speckle position in the image to be processed. The disparity of each pixel of each speckle position in the image to be processed collected by the sensor can be obtained. Compare with the preset speckle image to obtain the disparity corresponding to each pixel at each speckle position, and generate a corresponding disparity map. For any baseline, calculate the depth of each speckle based on the baseline and each pixel at each speckle position. When obtaining the depth value of each speckle, the corresponding depth value can be calculated based on the generated disparity map and the preset plane equation. Generate depth map. For example, the speckle image collected by the sensor and the reference plane speckle image recorded during the calibration stage are matched to calculate the disparity map; the depth map is calculated using the above disparity map, the reference plane equation, the camera focal length, the distance between the speckle and the projector .

Among them, for any baseline, the depth value of each pixel at each speckle position corresponding to the baseline is calculated based on the disparity corresponding to the baseline and each pixel at each speckle position. The calculation can be performed through a variety of preset algorithms, for example, Calculated through monocular depth algorithm. For example, Sensor and projector perform depth calculations in pairs: Sensor and projector 1 use the monocular depth calculation method to calculate the depth map m1 with S1 as the baseline; Sensor and projector 2 use the monocular depth calculation method with S2 as the baseline. Calculate the depth map m2; Sensor and projector 3 use the monocular depth calculation method to calculate the depth map m3 with S3 as the baseline. The specific calculation process can be found in subsequent embodiments.

In the embodiment of the present application, the inventor found that when calculating depth values based on different baselines, the best accuracy corresponding to different baselines is often a certain depth range, see Figure 3. Therefore, when calculating and generating a depth map based on each baseline, only the depth within a certain distance range is calculated for each baseline, thereby improving the accuracy of the calculation. Then the depth maps generated by each baseline are fused to obtain the target depth map, and a complete depth map is obtained through fusion.

In the existing technology, in order to improve the accuracy of depth calculation, multiple sensors will be installed. However, accessing a large number of image sensors not only places high requirements on SOC (System on Chip, system-on-chip), but also uses multiple SOCs. It will involve multiple SOC synchronization issues, resulting in high system complexity, which is not conducive to the real-time and problematic nature of the system, and also increases costs. Through the method of the embodiment of the present application, not only can the accuracy of depth calculation be improved through multiple projectors, but the projectors generally only need PWM (Pulse width modulation) control, and the circuit is simple and easy to control.

It can be seen that through the depth map generation system of the embodiment of the present application, different baselines representing the distances between multiple projectors and sensors can be used to calculate corresponding depth maps respectively, and the calculated depth maps can be fused. Thereby improving the accuracy of depth calculation.

Optionally, the processor is specifically configured to select a plane equation corresponding to any baseline; calculate each pixel at each speckle position according to the selected plane equation and the parallax corresponding to each pixel at each speckle position. depth, and obtain the depth value of each pixel at each speckle position.

In actual use, the sensor internal parameters can be calibrated in advance, and then the sensor internal parameters can be used to correct the image. For example, select a reference plane, and determine the plane equation of the reference plane in the camera coordinate system through the existing camera calibration algorithm, as well as the baseline distances s1, s2, and s3 between the projector and the sensor. The projector projects the speckle image to the reference plane, and records the reference images R1, R2, respectively, for each projector. R3 calculates the depth interval [0, d1), [d1, d2), [d2, d3), [d3, +∞), based on the maximum disparity calculation value dmax set by the algorithm. Among them, f is the equivalent focal length after correction of the camera, thereby achieving calibration between the sensor and the projector.

Among them, when calculating the depth of each speckle through plane equations and parallax, it is necessary to obtain and calculate the depth corresponding to different parallax based on the correspondence between the baseline, parallax, field of view, depth, etc. Therefore, during actual use, plane equations can be created in advance for different baselines, and the plane equations can represent the corresponding relationship between the parallax and depth corresponding to the baseline. When calculating depth, you can first select the corresponding plane equation based on the baseline, and then calculate the corresponding depth based on the parallax, thereby improving calculation efficiency and accuracy.

Optionally, the processor is specifically configured to calculate, for any baseline, the depth value of each pixel at multiple speckle locations based on the baseline and the disparity corresponding to each pixel at each speckle location; for any baseline, based on the parallax The preset depth range corresponding to the baseline, extract the depth value of the speckle whose depth value is within the preset depth range, and obtain the depth value of each pixel at each speckle position corresponding to the baseline, where each baseline corresponds to a preset Depth range, different baselines correspond to different preset depth ranges.

For example, Sensor and projector perform depth calculations in pairs: Sensor and projector 1 use the monocular depth calculation method to calculate the depth map m1 with S1 as the baseline; Sensor and projector 2 use the monocular depth calculation method with S2 as the baseline. Calculate the depth map m2; Sensor and projector 3 use the monocular depth calculation method to calculate the depth map m3 with S3 as the baseline. When performing fusion, m1 can be calculated with the minimum baseline s1 as a reference. The final depth map m is based on each point in m1 falling into the depth interval [0, d1), [d1, d2), [d2, d3), [d3,+∞) range determines which depth map to select the corresponding value from. The formula is as follows:

Finally, for any baseline, a depth map is generated based on the extracted brightness values.

Optional, multiple projectors, specifically used to project light through each projector alternately;

The sensor is specifically used to collect images to be processed at a preset interval.

Optionally, among the multiple projectors, the exposure time of the projector closest to the sensor is shorter than the exposure time of the projector farthest from the sensor.

In actual use, the exposure duration of different projectors can be directly proportional to the corresponding baseline. For example, taking three projectors as an example, the corresponding baselines are represented as d1, d2, and d3 respectively, and the corresponding exposure times of the three projectors are represented as t1, t2, and t3. You can calculate t1, t2, and t3 through the following steps:

If the time to synthesize one frame of the depth map is T, then t1+t2+t3=T;
t1/t2/t3=d1/d2/d3;

Then, t1=d1*T/(d1+d2+d3);
t2＝d2*T/(d1+d2+d3);
t3=d3*T/(d1+d2+d3).

Referring to Figure 4, the depth map generation system includes an image sensor, a plurality of projectors, a projector drive circuit, an image signal processing module (image signal processor), a depth image processing module, and a digital signal processing module (digital signal processor). Encoder, projector control module.

The projector control module is used to control multiple projectors to alternately encode the space. Each projector can correspond to an image signal processor respectively. The image encoded by projector 1 is input to the image signal processing module 1. The image encoded by projector 2 is input to the image signal processing module 2. The image encoded by projector 3 is input to the image signal processor. Signal processing module 3. Image signal processing module 1, image signal processing module 2, and image signal processing module 3 can be three physically independent modules, or the same module can be time-division multiplexed. In the same way, DPU (depth image processing module) 1, DPU2, DPU3, digital signal processor 1, and digital signal processor 2 can also be multiple physically independent modules, or the same module can be time-division multiplexed.

For the image acquisition process of three projectors, taking the output frame rate of 30FPS as an example, the total time to acquire image frames from three projectors is 1/30 seconds, and the average acquisition time of one frame is 1/90 seconds. Considering that the projection energy of the projector decreases with the distance, in order to make the far and near energy more uniform and the depth map quality to be higher, the exposure duration of the image frame of the short baseline projector is shorter than the exposure duration of the image frame of the long baseline projector. For example, the exposure time of projector image frame 1 is 5ms, the exposure time of projector image frame 2 is 10ms, and the exposure time of projector image frame 3 is 15ms.

Among them, in the embodiment of the present application, multiple projectors and sensors are used to form a multi-view system. This multi-view system can illuminate an object with infrared structured light through the projector and project artificial artifacts on the surface of the object in the absence of texture. Texture, and then use the dual-mode depth acquisition method to obtain the depth map. The depth map has higher quality and stronger reliability. Under normal circumstances, the distances s1 and s2 between the projector and the two cameras are smaller than the distance s3 between the two cameras. In an outdoor bright environment, the structure at a close distance The light intensity is high, and the monocular mode can be used to obtain the depth map. Due to the attenuation of structured light at long distances, the binocular depth calculation method can be used to obtain the fused depth map. Through the dual-mode depth acquisition method, the image quality obtained is compared with traditional structured light or The binocular solution is higher.

In order to illustrate the solution of the present application, refer to Figure 5 and the following description will be given in conjunction with specific embodiments:

Depth map generation method:

1. Offline processing:

Offline calibration between the sensor and the projector: select a reference plane, and determine the plane equation F of the reference plane in the camera coordinate system through the existing camera calibration algorithm, as well as the baseline distances s1, s2 between the projector and the sensor. s3.

The projector projects the speckle image to the reference plane, and each projector records the reference images R1, R2, and R3 respectively; according to the maximum parallax calculation value dmax set by the algorithm, the depth interval [0, d1), [d1, d2), [ d2,d3),[d3,+∞), where f is the equivalent focal length of the camera after correction.

2. Online processing:

Step 1: Laser speckle projector performs spatial encoding.

Step 2: The sensor collects images and preprocesses them. Preprocessing methods include contrast enhancement, histogram equalization, binarization and other methods. Preprocessing can increase image features and equalize brightness.

Step 3: Depth calculation is performed between the sensor and the projector in pairs: Sensor and projector 1 use the monocular depth calculation method to calculate the depth map m1 with S1 as the baseline, and Sensor and projector 2 use the monocular depth calculation method with S2 as the baseline. The depth map m2 is calculated from the baseline, and the Sensor and Projector 3 use the monocular depth calculation method to calculate the depth map m3 with S3 as the baseline.

The monocular depth calculation method is as follows: use the offline calibrated camera internal parameters to correct the image; perform image matching calculation on the speckle image obtained by the camera and the reference plane speckle image recorded during the calibration stage to obtain a disparity map; use the above disparity map, the reference plane equation and The depth map is calculated from the distance between the camera focal length, speckle, and projector.

Step 4: Perform a fusion operation on the three depth maps obtained in the previous step: m1 is calculated with the minimum baseline s1 as a reference, and the final depth map m is based on each point in m1 falling into the depth area determined during calibration. The range between [0,d1), [d1,d2), [d2,d3), [d3,+∞) determines which depth map to select the corresponding value from.

The second aspect of the embodiment of the present application provides a depth map generation method, which is applied to a processor in a depth map generation system. The depth map generation system includes: a sensor, a processor, and multiple projectors. See Figure 6, as mentioned above. Methods include:

Step S61: Obtain an image to be processed, where the image to be processed includes speckles after reflection of light projected by multiple projectors;

Step S62: Compare the image to be processed with the preset speckle image to obtain the parallax corresponding to each pixel at each speckle position in the image to be processed, where the preset speckle image includes the initial parallax of each pixel at each speckle position. reference position;

Step S63: For any baseline, calculate the depth value of each pixel at each speckle position corresponding to the baseline based on the disparity corresponding to the baseline and each pixel at each speckle position, where the baseline represents the distance between the projector and the sensor. Distance, the distance between different projectors and sensors is different;

Step S64: For any baseline, generate a depth map corresponding to the baseline based on the depth value of each pixel at each speckle position corresponding to the baseline;

Step S65: fuse the depth maps corresponding to each baseline to obtain a target depth map.

For any baseline, select the plane equation corresponding to the baseline;

According to the selected plane equation and the disparity corresponding to each pixel at each speckle position, the depth of each pixel at each speckle position is calculated to obtain the depth value of each pixel at each speckle position.

It can be seen that through the depth map generation method of the embodiment of the present application, multiple projectors and transmitters can be used Multiple baselines with different distances between sensors are used to generate corresponding depth maps respectively, and the generated depth maps are fused to improve the accuracy of depth calculation.

A third aspect of the embodiment of the present application provides an autonomous mobile device, including a depth map generation system. The depth map generation system includes: a sensor, a processor, and multiple projectors;

Multiple projectors for projecting light;

A sensor, used to collect images to be processed, where the images to be processed include speckles of light projected by multiple projectors after reflection;

A processor configured to compare the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel at each speckle position in the image to be processed, where the preset speckle image includes each pixel at each speckle position. the initial reference position; for any baseline, calculate the depth value of each pixel at each speckle position corresponding to the baseline according to the disparity corresponding to the baseline and each pixel at each speckle position, where the baseline represents the relationship between the projector and the sensor The distance between different projectors and sensors is different; for any baseline, a depth map corresponding to the baseline is generated based on the depth value of each pixel at each speckle position corresponding to the baseline; for the depth corresponding to each baseline The images are fused to obtain the target depth map.

Optionally, the processor is specifically configured to select a plane equation corresponding to any baseline; calculate the depth of each pixel at each speckle position based on the selected plane equation and the parallax corresponding to each pixel at each speckle position. , obtain the depth value of each pixel at each speckle position.

Optionally, the exposure time of the projector closest to the sensor among the multiple projectors is shorter than the exposure time of the projector farthest from the sensor.

Specifically, the above-mentioned autonomous mobile device may be a mobile robot, a sweeping robot, or other devices that can move automatically. The automatic mobile device can measure the depth and generate the depth map according to the depth map generation system, and then move or stop autonomously according to the detection results. For example, see Figure 7, the front and rear ends of the autonomous mobile device can each be equipped with multiple projector and a sensor that can be The processor controls the projectors at the front and rear to project speckles at the front and rear, and collects speckle images through the sensors at the front and rear, and then inputs the collected images into the processor for calculation, thereby obtaining the corresponding depth calculation results, such as distance Clustering of obstacles in front or behind, so that it stops when an obstacle is found in front through depth detection, or continues after the obstacle in front is removed through depth detection.

It can be seen that through the processor of the embodiment of the present application, multiple baselines with different distances between the projectors and sensors can be used to generate corresponding depth maps respectively, and the generated depth maps can be fused, thereby improving the efficiency of depth calculation. Accuracy.

The embodiment of the present application also provides an electronic device, as shown in Figure 8, including a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, the communication interface 802, and the memory 803 communicate through the communication bus 804. complete mutual communication,

Memory 803, used to store computer programs;

The processor 801 is used to execute the program stored in the memory 803 to implement the following steps:

Obtaining an image to be processed, wherein the image to be processed includes speckles after reflection of light projected by multiple projectors;

Compare the image to be processed with the preset speckle image to obtain the parallax corresponding to each pixel at each speckle position in the image to be processed, where the preset speckle image includes the initial reference position of each pixel at each speckle position;

For any baseline, calculate the depth value of each pixel at each speckle location corresponding to the baseline based on the disparity corresponding to the baseline and each pixel at each speckle location, where the baseline represents the distance between the projector and the sensor, which is different. The distance between the projector and sensor varies;

The depth maps corresponding to each baseline are fused to obtain the target depth map.

The communication bus mentioned in the above-mentioned electronic equipment can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the above-mentioned electronic devices and other devices.

The memory may include Random Access Memory (RAM) or To include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far away from the aforementioned processor.

The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (Network Processor, NP), etc.; it can also be a digital signal processor (Digital Signal Processor, DSP), special integrated Circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

In yet another embodiment provided by the present application, a computer-readable storage medium is also provided. A computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, any one of the above depth map generation is implemented. Method steps.

In yet another embodiment provided by this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the depth map generation methods in the above embodiments.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, e.g., the computer instructions may be transferred from a website, computer, server, or data center Transmission to another website, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), etc.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply There is no such actual relationship or sequence between these entities or operations. Furthermore, the terms "comprises,""comprises," or any other variations thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also those not expressly listed other elements, or elements inherent to the process, method, article or equipment. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article, or apparatus that includes the stated element.

Each embodiment in this specification is described in a related manner. The same and similar parts between the various embodiments can be referred to each other. Each embodiment focuses on its differences from other embodiments. In particular, for the method, autonomous mobile device, computer-readable storage medium and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For relevant details, please refer to the partial description of the method embodiments. .

The above descriptions are only preferred embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application are included in the protection scope of this application.

Claims

A depth map generation system including a sensor, a processor, and a plurality of projectors;

The plurality of projectors are used to project light;

The sensor is used to collect an image to be processed, wherein the image to be processed includes speckles after reflection of light projected by multiple projectors;

The processor is configured to compare the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel at each speckle position in the image to be processed, wherein the preset speckle image Including the initial reference position of each pixel at each speckle position; for any baseline, calculate the depth value of each pixel at each speckle position corresponding to the baseline based on the disparity corresponding to the baseline and each pixel at each speckle position. , wherein the baseline represents the distance between the projector and the sensor, and the distance between different projectors and the sensor is different; for any baseline, the depth of each pixel at each speckle position corresponding to the baseline value, generate a depth map corresponding to the baseline; fuse the depth maps corresponding to each baseline to obtain a target depth map.
The system of claim 1, wherein

The processor is specifically configured to calculate, for any baseline, the depth value of each pixel at multiple speckle locations based on the baseline and the parallax corresponding to each pixel at each speckle location; for any baseline, based on the parallax The preset depth range corresponding to the baseline, extract the depth value of the speckle whose depth value is within the preset depth range, and obtain the depth value of each pixel at each speckle position corresponding to the baseline, where each baseline corresponds to a preset Depth range, different baselines correspond to different preset depth ranges.
The system of claim 1, wherein

The processor is specifically configured to select a plane equation corresponding to any baseline; and calculate the depth of each pixel at each speckle position based on the selected plane equation and the parallax corresponding to each pixel at each speckle position. , obtain the depth value of each pixel at each speckle position.
The system of claim 1, wherein

The plurality of projectors are specifically used to alternately project light through each of the projectors;

The sensor is specifically used to collect images to be processed every preset time interval.
The system according to claim 1, characterized in that:

The exposure duration of the projector closest to the sensor among the plurality of projectors is shorter than the exposure duration of the projector farthest from the sensor.
A depth map generation method, applied to a processor in a depth map generation system, the depth map The generation system includes: a sensor, a processor and a plurality of projectors, and the method includes:

Obtaining an image to be processed, wherein the image to be processed includes reflected speckles of light projected by multiple projectors;

The image to be processed is compared with a preset speckle image to obtain the parallax corresponding to each pixel at each speckle position in the image to be processed, wherein the preset speckle image includes each pixel at each speckle position. The initial reference position of the pixel;

For any baseline, calculate the depth value of each pixel at each speckle position corresponding to the baseline based on the disparity corresponding to the baseline and each pixel at each speckle position, where the baseline represents the projector and the sensor The distance between different projectors and the sensor is different;

For any baseline, generate a depth map corresponding to the baseline based on the depth value of each pixel at each speckle position corresponding to the baseline;

The depth maps corresponding to each of the baselines are fused to obtain a target depth map.
The method according to claim 6, wherein for any baseline, the depth value of each pixel at each speckle position corresponding to the baseline is calculated based on the disparity corresponding to the baseline and each pixel at each speckle position. ,include:

For any baseline, calculate the depth values of each pixel at multiple speckle locations based on the disparity corresponding to the baseline and each pixel at each speckle location;

For any baseline, according to the preset depth range corresponding to the baseline, extract the depth value of the speckle whose depth value is within the preset depth range, and obtain the depth value of each pixel at each speckle position corresponding to the baseline, where, Each baseline corresponds to a preset depth range, and different baselines correspond to different preset depth ranges.
The method according to claim 6, wherein for any baseline, the depth value of each pixel at each speckle position corresponding to the baseline is calculated based on the disparity corresponding to the baseline and each pixel at each speckle position. ,include:

For any baseline, select the plane equation corresponding to the baseline;

According to the selected plane equation and the parallax corresponding to each pixel at each speckle position, the depth of each pixel at each speckle position is calculated to obtain the depth value of each pixel at each speckle position.
An autonomous mobile device includes a depth map generation system including: a sensor, a processor, and a plurality of projectors;

The plurality of projectors are used to project light;

The sensor is used to collect an image to be processed, wherein the image to be processed includes speckles after reflection of light projected by multiple projectors;

The processor is configured to compare the image to be processed with a preset speckle image to obtain the parallax corresponding to each pixel at each speckle position in the image to be processed, wherein the preset speckle image Including the initial reference position of each pixel at each speckle position; for any baseline, calculate the depth value of each pixel at each speckle position corresponding to the baseline based on the disparity corresponding to the baseline and each pixel at each speckle position. , wherein the baseline represents the distance between the projector and the sensor, and the distance between different projectors and the sensor is different; for any baseline, the depth of each pixel at each speckle position corresponding to the baseline value, generate a depth map corresponding to the baseline; fuse the depth maps corresponding to each baseline to obtain a target depth map.
The autonomous mobile device of claim 9, wherein

The processor is specifically configured to calculate, for any baseline, the depth value of each pixel at multiple speckle locations based on the baseline and the parallax corresponding to each pixel at each speckle location; for any baseline, based on the parallax The preset depth range corresponding to the baseline, extract the depth value of the speckle whose depth value is within the preset depth range, and obtain the depth value of each pixel at each speckle position corresponding to the baseline, where each baseline corresponds to a preset Depth range, different baselines correspond to different preset depth ranges.
The autonomous mobile device of claim 9, wherein

The processor is specifically configured to select a plane equation corresponding to any baseline; and calculate the depth of each pixel at each speckle position based on the selected plane equation and the parallax corresponding to each pixel at each speckle position. , obtain the depth value of each pixel at each speckle position.
The autonomous mobile device of claim 9, wherein

The plurality of projectors are specifically used to alternately project light through each of the projectors;

The sensor is specifically used to collect images to be processed every preset time interval.
The autonomous mobile device according to claim 9, characterized in that

The exposure duration of the projector closest to the sensor among the plurality of projectors is shorter than the exposure duration of the projector farthest from the sensor.
A computer-readable storage medium. A computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the method steps described in any one of claims 6-8 are implemented.