WO2021114497A1

WO2021114497A1 - Time-of-flight-based depth camera and three-dimensional imaging method

Info

Publication number: WO2021114497A1
Application number: PCT/CN2020/077847
Authority: WO
Inventors: 孙瑞; 周兴; 王兆民; 孙飞; 马宣; 杨神武; 方兆翔
Original assignee: 深圳奥比中光科技有限公司
Priority date: 2019-12-12
Filing date: 2020-03-04
Publication date: 2021-06-17
Also published as: CN111025329A

Abstract

A time-of-flight-based depth camera (100) and a three-dimensional imaging method. The depth camera (100) comprises: a transmitting module (101) comprising at least two light sources for transmitting an optical signal, the optical signal being provided with a certain field of view; a receiving module (102), comprising an image processor consisting of at least one pixel, for acquiring a reflected optical signal; a control and processing module (103) for controlling the transmitting module (101) and the receiving module (102) and processing an optical signal reflected back by a full field of view region (104), and determining a target region (106) where a target object (105) is located according to the optical signal, so as to control the transmitting module (101) to transmit an optical signal only to the target region (106) and control the receiving module (102) to acquire the optical signal reflected back by the target region(106) and process same to obtain depth information of the target object (105), the transmitting module (101) having: a first type of illumination which refers to transmitting an optical signal to the full field of view region (104) for scanning, and a second type of illumination which refers to transmitting an optical signal only to the target region (106) for scanning. The depth camera and the three-dimensional imaging method can reduce the power consumption of a light source, reduce the power consumption of depth calculation, and increase the calculation rate.

Description

A depth camera and three-dimensional imaging method based on flight time

Technical field

The present invention relates to the field of optics and electronics technology, in particular to a depth camera and a three-dimensional imaging method based on flight time.

Background technique

Based on the time-of-flight (TOF) three-dimensional imaging technology, the vertical cavity surface laser transmitter (VCSEL) illuminates the target surface with a certain field of view (FOV). When irradiating with a full field of view at the same time, in order to improve the signal-to-noise ratio and the accuracy of ranging and imaging, it is often necessary to increase the peak power of the light source, and the power consumption is relatively large, which increases the difficulty of designing the power supply system and the heat dissipation system. Due to the FOV range, the target may exist in a very small area or part of the area, and the rest of the FOV data is not necessary, which consumes a lot of extra work such as light source and depth calculations, and these redundant work Will increase a lot of power consumption and calculation speed. Therefore, how to increase the depth calculation rate and reduce power consumption is a problem that needs to be solved in the TOF system design, especially for the indirect time-of-flight ranging system (iTOF).

Summary of the invention

In order to solve the existing technical problems, the present invention provides a depth camera and a three-dimensional imaging method based on flight time.

In order to solve the above problems, the technical solutions adopted by the present invention are as follows:

A depth camera based on time of flight, including a transmitting module, a receiving module, and a control and processing module, wherein: the transmitting module includes at least two light sources for transmitting light signals, and the light signals are set with a certain angle of view The receiving module includes an image processor composed of at least one pixel, which is used to collect the light signal reflected by the full field of view area and the target area; the control and processing module is used to control the transmitting module and the The receiving module, and processing the light signal reflected back through the full field of view area, determining the target area where the target object is located according to the light signal, and controlling the transmitting module to only emit light signals to the target area; controlling the receiving module to collect the light signal The light signal reflected back from the target area is processed to obtain the depth information of the target object; wherein, the transmitting module has a first type of lighting and a second type of lighting, and the first type of lighting is It refers to scanning the light signal emitted from the full field of view area, and the second type of illumination refers to scanning only the light signal emitted from the target area where the target object is located.

In an embodiment of the present invention, the emission module includes: a first light source, a second light source, and a third light source for emitting light signals outward; a first diffuser, a second diffuser, and a third diffuser. The diffuser corresponds to the first light source, the second light source, and the third light source, respectively, and is used to modulate the field of view angle of the optical signal.

In yet another embodiment of the present invention, the emission module includes: a first light source, a second light source, and a third light source for emitting light signals outward; a first liquid crystal modulation element, a second liquid crystal modulation element, and a second light source; Three liquid crystal modulation elements, respectively corresponding to the first light source, the second light source, and the third light source, are used to control the divergence angle of the optical signal.

In still another embodiment of the present invention, the emission module includes: a substrate for providing at least three oblique angles to carry the light source; the first light source, the second light source, and the third light source are placed at different oblique angles On the substrate, it is used to emit light signals to the outside. The base is semicircular or isosceles trapezoid. The first type of lighting includes turning on the first light source, the second light source, and the third light source at the same time; the second type of lighting includes: turning on only the first light source; or, turning on only the second light source. Two light sources; or, only start the third light source; or, start the first light source and the second light source at the same time; or; start the second light source and the third light source at the same time; or, start all at the same time The first light source, the second light source, and the third light source.

The present invention also provides a three-dimensional imaging method, including the following steps: S1: use the first type of illumination to obtain a first frame image of the full field of view area; S2: determine the target area where the target object is located according to the first frame image; S3 : Use the second type of lighting to illuminate the target area; S4: Obtain a second frame of image to obtain the depth information of the target.

In an embodiment of the present invention, the first type of illumination refers to turning on all light sources to scan the full field of view area; the second type of illumination refers to turning on only the light source corresponding to the target area to scan the entire field of view. The target area is scanned.

In another embodiment of the present invention, the first type of illumination includes using an infrared light source to scan the full field of view area to obtain the first frame of depth image.

In still another embodiment of the present invention, the first type of illumination includes using an RGB sensor to collect a two-dimensional image of the full field of view area under ambient light or LED illumination of the full field of view area as the source of light. Describe the first frame of image.

The beneficial effect of the present invention is to provide a depth camera and a three-dimensional imaging method based on time of flight, by setting multiple light sources to illuminate different areas respectively, and defining the area where the target object is located by roughly scanning the image obtained by scanning the full field of view area, Then only the light source corresponding to the defined target area is turned on to perform a fine scan on the target to obtain a depth map, thereby achieving the effect of reducing the power consumption of the light source, reducing the power consumption of depth calculation, reducing the difficulty of heat dissipation design, and improving the calculation rate.

Description of the drawings

Fig. 1 is a schematic diagram of a depth camera based on time-of-flight according to the present invention.

Fig. 2 is a schematic structural diagram of a transmitting module according to the present invention.

Fig. 3 is a schematic structural diagram of another transmitting module provided according to the present invention.

Fig. 4 is a structural diagram of another transmitting module provided according to the present invention.

Fig. 5 is a schematic diagram of a three-dimensional imaging method based on adaptive illumination of a time-of-flight depth camera according to the present invention.

Among them, 100-depth camera, 101-transmitting module, 102-receiving module, 103-control and processing module, 104-full field of view area, 105-target object, 106-target area, 11-optical signal, 12-optical signal , 200-emission module, 201-first light source, 202-second light source, 203-third light source, 204-first diffuser, 205-second diffuser, 206-third diffuser, 300- Transmitting module, 301-first light source, 302-second light source, 303-third light source, 304-first liquid crystal modulation element, 305-second liquid crystal modulation element, 306-third liquid crystal modulation element, 400-transmitting module, 401-substrate, 402-first light source, 403-second light source, 404-third light source.

Detailed ways

In order to make the technical problems, technical solutions, and beneficial effects to be solved by the embodiments of the present invention clearer, the following further describes the present invention in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, but not used to limit the present invention.

It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, the connection can be used for fixing or circuit connection.

It should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top" The orientation or positional relationship indicated by "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the embodiments of the present invention and simplifying the description, rather than indicating or implying what is meant. The device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention.

In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" may explicitly or implicitly include one or more of these features. In the description of the embodiments of the present invention, "a plurality of" means two or more than two, unless otherwise specifically defined.

Fig. 1 is a schematic diagram of a depth camera based on time-of-flight according to the present invention. The depth camera 100 is a TOF depth camera, and includes a transmitting module 101, a receiving module 102, and a control and processing module 103. The control and processing module 103 is connected to the transmitting module 101 and the receiving module 102. Among them, the transmitting module 101 includes at least two light sources for emitting light signals (such as pulsed laser beams) to the full-field area 104; the receiving module 102 includes an image processor composed of at least one pixel to capture the entire field-of-view area. 104 The light signal reflected back; the control and processing module 103 is used to control the transmitting module 101 and the receiving module 102, and process the light signal reflected by the full field of view area 104, and determine the target area of the target 105 according to the reflected light signal 106 and then control the emission module 101 to only emit light signals to the target area 106 where the target 105 is located; wherein, the emission module has a first type of lighting and a second type of lighting, and the first type of lighting refers to emitting light to the area within the full field of view. The light signal is uniformly illuminated, and the second type of illumination means that the light signal is emitted only to the area where the target object is located. It should be understood that the depth camera 100 may also include a circuit module, a power module, a housing, and other components, which are not fully shown here. The depth camera 100 may be set independently or integrated into electronic devices such as mobile phones, tablets, computers, etc., and there is no limitation here.

In an embodiment, the light source of the emission module 101 is set to any suitable light source as required. For example, it can be a single light source, such as a side-emitting laser, or an array composed of multiple light sources, such as a light source chip array composed of multiple vertical cavity surface lasers, or a side-emitting laser diode, infrared laser, LED, etc. Set according to specific needs, there is no restriction here.

In one embodiment, the image sensor of the receiving module 102 is an image sensor specially used for time-of-flight (TOF) measurement. For example, it may be CMOS (Complementary Metal Oxide Semiconductor), APD (Avalanche Photodiode), SPAD ( Single-photon avalanche photodiodes) and other image sensors. The pixels of the image sensor can be in the form of single dot, linear array or area array.

In one embodiment, the control and processing module 103 is connected to the transmitting module 101 and the receiving module 102 respectively, and is used to send control signals to each module to implement corresponding control operations, and perform related calculations or processing on the received optical signals. Wait. The functions of the control and processing module 103 include providing periodic modulation signals required by the transmitter module 101 to emit optical signals, and providing acquisition signals for the receiving module 102 to collect optical signals. It can also provide auxiliary monitoring signals, including temperature sensing, overcurrent , Overvoltage protection, drop-off protection, etc.

In one embodiment, the control and processing module 103 sends a control signal to the transmitting module 101. The transmitting module 101 uses the first type of illumination, turns on all the light sources to perform a rough scan of the full field of view area 104, sends the acquisition signal to the receiving module 102, and receives The module 102 collects the light signal reflected back by the full field of view area 104, the control and processing module 103 performs corresponding processing on the collected light signal, determines the target area 106 of the target object 105, and sends a control signal to the transmitting module 101 again, The transmitting module 101 uses the second type of illumination, turning on part of the light source only emits the light signal 11 to the target area 106 of the target 105, and performs fine scanning, while the receiving module 102 collects the light signal reflected back through the target area under the action of the collected signal 12. The control and processing module 103 processes the collected optical signal 12 to obtain the depth information of the target 105.

Fig. 2 is a structural diagram of a transmitting module provided according to the present invention. The emission module 200 includes a first light source 201, a second light source 202, a third light source 203, a first diffuser 204, a second diffuser 205 and a third diffuser 206. The first light source 201, the second light source 202, and the third light source 203 are used to emit light signals outward. The surfaces of the first diffuser 204, the second diffuser 205 and the third diffuser 206 have micro-nano structures, which can be According to different requirements, by changing the size, period, and shape of the micro-nano structure, the field angle of the emitted light signal is modulated, so that the first light source 201, the second light source 202, and the third light source 203 have a certain field angle.

Fig. 3 is a structural diagram of another transmitting module provided according to the present invention. The emission module 300 includes a first light source 301, a second light source 302, a third light source 303, a first liquid crystal modulation element 304, a second liquid crystal modulation element 305, and a third liquid crystal modulation element 306. The first light source 301, the second light source 302, and the third light source 303 are used to emit light signals outward. The first liquid crystal modulation element 304, the second liquid crystal modulation element 305, and the third liquid crystal modulation element 306 are controlled by different voltages. The liquid crystal can be deflected at different angles, and then the divergence angle of the emitted light signal can be controlled, so that the first light source 301, the second light source 302, and the third light source 303 have different fields of view, respectively.

In one embodiment, the first diffuser 204, the second diffuser 205, the third diffuser 206, the first liquid crystal modulation element 304, the second liquid crystal modulation element 305, and the third liquid crystal modulation element 306 may be Free combination, used to modulate the field of view of the light signal emitted by the light source.

Fig. 4 is a structural diagram of another transmitting module provided according to the present invention. The emission module 400 includes a substrate 401, a first light source 402, a second light source 403, and a third light source 404. The first light source 402, the second light source 403, and the third light source 404 are used to emit light signals outward, and the substrate 401 is set in an isosceles trapezoid shape, so that the first light source 402, the second light source 403, and the third light source 404 are placed at different tilt angles Therefore, the optical signals emitted by the first light source 402, the second light source 403, and the third light source 404 have different fields of view. It should be understood that the shape of the substrate 401 may also be a semicircle, which can provide at least three angles to place the light source, and the angle setting can be adjusted according to specific requirements, and there is no limitation here.

In one embodiment, the control and processing module sends a control signal to the emission module to turn on the first light source 201, the second light source 202, and the third light source 203 to emit light signals to the full field of view area to perform a rough scan of the full field of view area, The receiving module collects the light signal reflected from the full field of view area, and the control and processing module processes the light signal and judges the target area where the target object is located. When the control and processing module again emits light signals to the target area where the target object is located, if the target object is within the field of view of the first light source 201 emitting light signals, only the first light source 201 is turned on to emit light signals to the target area where the target object is located. The second light source 202 and the third light source 203 are not turned on at this time; if the target is within the field of view of the second light source 202 emitting light signals, only the second light source 202 is turned on to emit light signals to the target area where the target is located, and the first light source 201 and the third light source 203 are not turned on; if the target is in the field of view of the third light source 203 emitting light signals, only the third light source 203 is turned on to emit light signals to the target area where the target is located, the first light source 201 and the second light source 203 is not turned on; if the target occupies the field of view of the first light source 201 and the second light source 202 at the same time, the first light source 201 and the second light source 202 are turned on at the same time, and the third light source 203 is not turned on; if the target occupies the second light source at the same time In the field of view of 202 and the third light source 203, the first light source 201 is not turned on; if the target occupies the field of view of the first light source 201, the second light source 202, and the third light source 203 at the same time, the three light sources are turned on at the same time.

It should be noted that the emission module may also include two light sources, four light sources, five light sources, etc. This embodiment only uses three light sources for description, but is not limited thereto. By setting multiple light sources, the power consumption of the light source and the depth calculation power consumption can be greatly reduced, and the subsequent depth calculation rate can be increased. For example, the power of using one light source to illuminate the full field of view area is 2.5W. According to the present invention, multiple light sources are set according to the target area. Turn on the corresponding light source, you can select two light sources with a power of 1.25W or three powers of 0.83W, so that the corresponding light source is turned on for lighting in the area where the target is located, which significantly reduces the power consumption.

Fig. 5 is a three-dimensional imaging method based on adaptive illumination of a time-of-flight depth camera according to the present invention. Explained in four steps:

Step S1: Use the first type of illumination to obtain the first frame of image of the full field of view area;

Step S2: Judging the target area where the target object is located according to the first frame of image;

Step S3: Use the second type of lighting to illuminate the target area;

Step S4: Obtain a second frame of image to obtain the depth information of the target object.

More specifically, in step S1, the first type of illumination means that all light sources are turned on to roughly scan the full field of view area, and the receiving module collects the light signal reflected back through the full field of view area, and then processes the light signal based on the control and processing module. The first frame image of the full field of view area; in step S2, determine the area where the target object is located according to the obtained first frame image, and define the area where the target object is located; in step S3, according to the target area defined in step S2, use the second Type lighting performs a fine scan on the defined target area, where the second type of lighting means that under the control of the control and processing module, only the light source corresponding to the target area defined in step S2 is turned on; in step S4, in the second In the case of type lighting, the receiving module collects the light signal reflected back through the defined target area to obtain the second frame of image, which is processed by the control and processing module to obtain the depth information of the target, which improves the scanning accuracy of the target. It should be understood that the size of the target area is specifically determined according to the size of the target object.

In one embodiment, the first type of illumination includes using an infrared light source to illuminate the entire field of view area, perform a rough scan to obtain the first frame of depth map, define the area where the target is located according to the first frame of depth map, and then turn on the define target The corresponding light source in the area where the object is located is finely scanned and processed to obtain the depth information of the object.

In one embodiment, the first type of illumination includes the use of RGB sensors to collect two-dimensional images of the full field of view under ambient light or LED illumination of the full field of view, and determine the area where the target object is located through the collected two-dimensional images , And then turn on the light source corresponding to the target area to perform fine scanning of the target, and obtain the depth information of the target after processing.

The beneficial effects achieved by the present invention are: setting a plurality of light sources to illuminate different areas respectively, defining the area where the target object is located by roughly scanning the image obtained from the full field of view area, and then only turning on the light source corresponding to the defined target area to target the target object. Perform a fine scan to obtain a depth map, thereby achieving the effect of reducing the power consumption of the light source, reducing the power consumption of depth calculation, reducing the difficulty of heat dissipation design, and improving the calculation rate.

A person of ordinary skill in the art can understand that all or part of the steps in the above embodiments can be implemented by hardware, or by a program to instruct relevant hardware. The program can be stored in a computer-readable storage medium. The storage medium mentioned can be a read-only memory, a magnetic disk or an optical disk, etc.

The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments, and it cannot be considered that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art to which the present invention belongs, without departing from the concept of the present invention, several equivalent substitutions or obvious modifications can be made, and the same performance or use should be regarded as belonging to the protection scope of the present invention.

Claims

A depth camera based on time-of-flight, which is characterized by comprising a transmitting module, a receiving module, and a control and processing module, wherein:

The transmitting module includes at least two light sources for transmitting optical signals, and the optical signals are set with a certain angle of view;

The receiving module includes an image processor composed of at least one pixel, which is used to collect the light signal reflected back through the full field of view area and the target area;

The control and processing module is used to control the transmitting module and the receiving module, and process the light signal reflected back through the full field of view area, and determine the target area where the target is located according to the light signal to control the transmitting module Only emit light signals to the target area; control the receiving module to collect the light signals reflected back from the target area, and process the light signals to obtain the depth information of the target;

Wherein, the emission module has a first type of illumination and a second type of illumination, the first type of illumination refers to scanning the light signal emitted from the full field of view area, and the second type of illumination refers to only the The target area where the target is located emits light signals for scanning.
The depth camera based on time of flight of claim 1, wherein the transmitting module comprises:

The first light source, the second light source, and the third light source are used to emit light signals outward;

The first diffuser, the second diffuser, and the third diffuser correspond to the first light source, the second light source, and the third light source, respectively, and are used to adjust the angle of view of the optical signal Make modulation.
The depth camera based on time of flight of claim 1, wherein the transmitting module comprises:

The first light source, the second light source, and the third light source are used to emit light signals outward;

The first liquid crystal modulation element, the second liquid crystal modulation element and the third liquid crystal modulation element respectively correspond to the first light source, the second light source and the third light source, and are used for controlling the divergence angle of the optical signal.
The depth camera based on time of flight of claim 1, wherein the transmitting module comprises:

The substrate is used to provide at least three oblique angles to carry the light source;

The first light source, the second light source, and the third light source are placed on the substrate at different inclination angles for emitting light signals outward.
The time-of-flight-based depth camera of claim 4, wherein the base is semicircular or isosceles trapezoid.
5. The time-of-flight-based depth camera according to any one of claims 2-5, wherein the first type of illumination comprises simultaneously turning on the first light source, the second light source, and the third light source;

The second type of illumination includes: only the first light source is turned on; or, only the second light source is turned on; or, only the third light source is started; or, the first light source and the second light source are started simultaneously Light source; or; start the second light source and the third light source at the same time; or, start the first light source, the second light source, and the third light source at the same time.
A three-dimensional imaging method is characterized in that it comprises the following steps:

S1: Use the first type of illumination to obtain the first image of the full field of view area;

S2: Judging the target area where the target object is located according to the first frame of image;

S3: Use the second type of lighting to illuminate the target area;

S4: Obtain a second frame of image to obtain depth information of the target object.
The three-dimensional imaging method according to claim 7, wherein the first type of illumination refers to turning on all light sources to scan the full field of view area; the second type of illumination refers to turning on only the target area The corresponding light source scans the target area.
8. The three-dimensional imaging method of claim 7, wherein the first type of illumination comprises scanning the full field of view area with an infrared light source to obtain the first frame of depth image.
The three-dimensional imaging method according to claim 7, wherein the first type of illumination includes using an RGB sensor to collect the second type of illumination of the full field of view area under ambient light or LED illumination of the full field of view area. A two-dimensional image is used as the first frame image.