WO2021008209A1

WO2021008209A1 - Depth measurement apparatus and distance measurement method

Info

Publication number: WO2021008209A1
Application number: PCT/CN2020/089758
Authority: WO
Inventors: 王兆民; 武万多
Original assignee: 深圳奥比中光科技有限公司
Priority date: 2019-07-12
Filing date: 2020-05-12
Publication date: 2021-01-21
Also published as: CN110333501A

Abstract

Disclosed are a depth measurement apparatus and a distance measurement method. The depth measurement apparatus (10, 40) comprises: a light emission module (11, 41), which includes a light source (111) and a patterned optical element (112), wherein the light source (111) is used for emitting a light beam, the amplitude of which is modulated, on a time sequence, and the patterned optical element (112) is used for receiving the light beam and then emitting a structured light beam to a target object (20); a TOF image sensor (121), which includes at least one pixel (51), wherein the pixel (51) is used for receiving the structured light beam reflected by the target object (20) and forming an electrical signal; and a control and processing circuit (13, 43), which receives the electrical signal, calculates intensity information of the reflected structured light beam to form a structured light pattern, and calculates a depth image of the target object (20) by using the structured light pattern. The TOF image sensor (121) is combined with the structured light source, amplitude modulation is used at an emission end, and multi-tap pixel collection is used at a receiving end, thereby achieving a depth measurement method, which is hard to achieve by means of a traditional solution, capable of resisting ambient light interference.

Description

Depth measuring device and distance measuring method

Technical field

The present invention relates to the field of optical measurement technology, in particular to a depth measurement device and a distance measurement method.

Background technique

The full name of ToF is Time-of-Flight, that is, time of flight. ToF ranging technology is a technology that achieves precise ranging by measuring the round-trip flight time of light pulses between the transmitting/receiving device and the target object. In the ToF technology, the technology of directly measuring the optical time of flight is called dToF (direct-TOF); the emitted light signal is periodically modulated, and the phase delay of the reflected light signal relative to the emitted light signal is measured. The measurement technique that calculates the time of flight phase delay is called iToF (Indirect-TOF) technique. According to the different types of modulation and demodulation, it can be divided into continuous wave (CW) modulation and demodulation method and pulse modulation (Pulse Modulated, PM) modulation and demodulation method.

The structured light ranging technology emits a structured light beam to a space object, and then collects the structured light pattern formed by the structured light beam modulated and reflected by the object, and finally uses the triangulation method to perform depth calculation to obtain the depth data of the object. Commonly used structured light patterns include irregular spot patterns, stripe patterns, phase shift patterns and so on.

In contrast, TOF technology does not require complex image processing calculations (such as structured light image matching calculations), and can maintain relatively high measurement accuracy at medium and long distances; while structured light technology has very high measurement accuracy at close distances. Accuracy, but the accuracy will increase at least linearly as the distance increases. In addition, because structured light technology calculates depth by collecting images reflecting the intensity of reflected light, it will inevitably be affected by ambient light on light intensity. Relatively speaking, TOF is superior to structured light technology in terms of anti-interference of ambient light. .

Summary of the invention

In order to solve the existing problems, the present invention provides a depth measuring device and a distance measuring method.

In order to solve at least one of the above problems, the technical solutions adopted by the present invention are as follows:

The depth measuring device includes: a light emitting module, including a light source and a patterned optical element, the light source is used to emit a light beam whose amplitude is modulated in time sequence, and the patterned optical element is used to receive the light beam and emit to a target object Structured light beam; TOF image sensor, including at least one pixel, the pixel is used to receive the structured light beam reflected by the target object and form an electrical signal; a control and processing circuit that receives the electrical signal and calculates the reflection The intensity information of the structured light beam is used to form a structured light pattern, and the structured light pattern is used to calculate a depth image of the target object.

In an embodiment of the present invention, the structured light beam includes an irregular spot pattern beam, a stripe pattern beam, and a two-dimensional coded patterned beam; the amplitude of the beam corresponding to each spot in the irregular spot pattern beam is The timing is modulated by continuous wave, square wave or pulse.

In another embodiment of the present invention, the pixel includes at least 3 taps, and the taps are used to separately collect the electrical signals generated by the structured light beam reflected by the target object within a single frame period. The light emitting module emits a light beam whose amplitude is modulated by a sine wave in time sequence. Each pixel of the TOF image sensor includes at least 4 taps, which are used to collect and convert the light signal 4 times in a single frame period. The electrical signals C1, C2, C3 and C4 are collected at the same time and interval for 4 times; the weighted average method is used to calculate the intensity B of the structured light beam, as follows:

Or, calculate the intensity B of the structured light beam by adding an ambient light elimination factor, as follows:

The light emitting module emits a light beam whose amplitude is pulse-modulated in time sequence, and each pixel of the TOF image sensor includes at least 3 taps, which are respectively used to collect and convert the optical signal three times in a single frame period into an electrical signal C1 , C2 and C3, the trigger time of the first tap is synchronized with the emission time of the light pulse, and the trigger duration is the same as the pulse width. The second tap and the third tap are triggered successively at the end of the first tap, and the trigger duration is equal to The first taps are the same; the weighted average method is used to calculate the intensity B of the structured light beam, specifically as follows:

Or, the intensity B of the structured light beam is calculated by adding an ambient light elimination factor, specifically as follows: B=C ₁ +C ₂ -2C ₃ .

The present invention also provides a distance measurement method, including: using a light source to emit a light beam whose amplitude is modulated in time sequence, and the light beam is received by a patterned optical element to form a structured light beam and emitted to a target object; using a TOF image containing at least one pixel The pixels of the sensor receive the structured light beam reflected by the target object and form an electrical signal; receive the electrical signal and calculate the intensity information of the reflected structured light beam to form a structured light pattern, and use the The structured light pattern calculates the depth image of the target object.

In another embodiment of the present invention, the pixel includes at least 3 taps, and the taps are used to separately collect electrical signals generated by the structured light beam reflected by the target object within a single frame period. The structured light beam is a light beam whose amplitude is modulated by a sine wave in time series, and each pixel includes at least 4 taps, which are respectively used to collect 4 optical signals and convert them into the electrical signals C1 and C2 in a single frame period. , C3 and C4, the time and interval of the 4 acquisitions are the same; the weighted average method is used to calculate the intensity B of the structured light beam, as follows:

The structured light beam is a light beam whose amplitude is pulse-modulated in time series. Each pixel includes at least 3 taps, which are respectively used to collect and convert optical signals three times in a single frame period into electrical signals C1, C2, and C3, The trigger time of the first tap is synchronized with the emission time of the light pulse, and the trigger duration is the same as the pulse width. The second tap and the third tap are triggered one after another at the end of the first tap, and the trigger duration is the same as that of the first tap. Same; the weighted average method is used to calculate the intensity B of the structured light beam, as follows:

The beneficial effects of the present invention are: a depth measurement device and a distance measurement method are provided, the TOF image sensor is combined with a structured light source, the transmitting end adopts amplitude modulation, and the receiving end adopts multi-tap pixel collection, which can make the The invented device and method have more functions than the traditional solution, and realize a depth measurement method against ambient light interference that is difficult to achieve with the traditional solution.

Description of the drawings

Fig. 1 is a schematic structural diagram of a depth measuring device in an embodiment of the present invention.

Fig. 2 is a schematic diagram of the principle of a depth measuring device in an embodiment of the present invention.

Fig. 3 is a schematic diagram of the principle of another depth measuring device in an embodiment of the present invention.

Fig. 4 is a schematic structural diagram of a depth measurement device integrating TOF and structured light solutions in an embodiment of the present invention.

FIG. 5 is a schematic structural diagram of another depth measurement device that integrates TOF and structured light solutions in an embodiment of the present invention.

Fig. 6 is a schematic diagram of structured light and TOF timing control in an embodiment of the present invention.

FIG. 7 is a schematic diagram of the control and processing circuit structure in an embodiment of the present invention.

Fig. 8 is a schematic diagram of a distance measurement method in an embodiment of the present invention.

Fig. 9 is a schematic diagram of another distance measurement method in an embodiment of the present invention.

Fig. 10 is a schematic diagram of a method for processing the electrical signal in an embodiment of the present invention.

Fig. 11 is a schematic diagram of another method for processing the electrical signal in an embodiment of the present invention.

Among them, 10-depth measuring device, 11-light emission module, 12-acquisition module, 13-control and processing circuit, 14- flood light source module, 111-light source, 112-patterned optical element, 121-TOF Image sensor, 122-lens unit, 20-target object, 30-emission beam, 40-reflected beam, 201-spot structured light patterned beam, 202-spot, 40-depth measuring device, 41-light emitting module, 42 -Acquisition module, 43- control and processing circuit, 44- flood light source module, 51- pixel, 52- spot, 70- chip, 71- phase calculation module, 72- calibration module, 73- first post-processing module , 74-1-splitter, 74-2-multiplexer, 75-amplitude calculation module, 76-pre-processing module, 77-matching module, 78-second post-processing module, 79-fusion module.

Detailed ways

In order to make the technical problems, technical solutions, and beneficial effects to be solved by the embodiments of the present invention clearer and clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention.

It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, the connection can be used for fixing or circuit connection.

It should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top" , "Bottom", "Inner", "Outer", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, rather than indicating or implying. The device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention.

In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" may explicitly or implicitly include one or more of these features. In the description of the embodiments of the present invention, "a plurality of" means two or more than two, unless otherwise specifically defined.

Fig. 1 is a depth measuring device according to an embodiment of the present invention. The depth measuring device 10 includes a light emitting module 11, a collection module 12, and a control and processing circuit 13. The light emitting module 11 is used to emit a light beam 30 to a target object 20, and the light beam 30 is amplitude modulated in time series. A structured light beam that reaches the target space to illuminate the target object 20 in the space. At least part of the emitted light beam 30 is reflected by the target object 20 to form a reflected light beam 40, and at least part of the reflected light beam 40 is collected by the collection module 12, The control and processing circuit 13 is respectively connected with the light emitting module 11 and the collection module 12 to control the emission and collection of the light beam, and at the same time receive the electric signal collected from the collection module 12, and calculate the electric signal to obtain Depth information of the target object.

The light emitting module 11 includes a light source 111, a patterned optical element 112, a light source driver (not shown in the figure), and the like. The light source 111 can be a light source such as a light emitting diode (LED), an edge emitting laser (EEL), a vertical cavity surface emitting laser (VCSEL), or a light source array composed of multiple light sources. The light beam emitted by the light source can be visible light or infrared light. , UV light, etc. The light source 111 is controlled by the light source driver (which can be further controlled by the control and processing circuit 13) to emit light beams outward after being modulated with a certain timing amplitude. For example, in one embodiment, the light source 111 emits light at a certain frequency under control. Pulse modulated beam, square wave modulated beam, sine wave modulated beam and other beams. It is understandable that a part of the control and processing circuit 13 or a sub-circuit independent of the control and processing circuit 13 can be used to control the light source 111 to emit related light beams, such as a pulse signal generator.

The patterned optical element 112 receives the light beam from the light source 111 and emits structured light beams, such as irregularly arranged spot patterned beams, striped patterned beams, and two-dimensional coded patterned beams. In some embodiments, the patterned optical element 112 is also used to expand the received light beam to expand the angle of view. It can be understood that the amplitude of the light beam modulated by the patterned optical element 112 is still modulated in a certain time sequence, that is, the incident sine wave modulated light beam still emits the sine wave modulated light beam. In an embodiment of the present invention, the amplitude of the beam corresponding to each spot in the irregular spot pattern beam is modulated in a continuous wave, square wave or pulse manner in time sequence.

The acquisition module 12 includes a TOF image sensor 121, a lens unit 122, and may also include a filter (not shown in the figure). The lens unit 122 receives and reflects at least part of the structured light beam from the target object and images it on at least part of the TOF. On the image sensor 121, the filter needs to select a narrow-band filter that matches the wavelength of the light source to suppress background light noise in the remaining wavelength bands. The TOF image sensor 121 can be an image sensor composed of charge coupled devices (CCD), complementary metal oxide semiconductors (CMOS), avalanche diodes (AD), single photon avalanche diodes (SPAD), etc. The size of the array represents the resolution of the depth camera Rate, such as 320x240, etc. Generally, connected to the image sensor 121 also includes a readout circuit composed of one or more of a signal amplifier, a time-to-digital converter (TDC), an analog-to-digital converter (ADC) and other devices (not shown in the figure). ).

Generally, the TOF image sensor 121 includes at least one pixel. Compared with a traditional image sensor that is only used for taking pictures, each pixel here includes more than two taps (tap, used to store and read under the control of the corresponding electrode). Take or discharge the charge signal generated by incident photons), for example, including 3 taps, switch the taps in a certain order within a single frame period (or single exposure time) to collect the corresponding photons for receiving light signals And converted into electrical signals.

The control and processing circuit 13 can be an independent dedicated circuit, such as a dedicated SOC chip, FPGA chip, ASIC chip, etc. composed of CPU, memory, bus, etc., or a general processing circuit, such as when the depth camera is integrated into In smart terminals such as mobile phones, TVs, and computers, the processing circuit in the terminal can be used as at least a part of the control and processing circuit 13.

The control and processing circuit 13 is used to provide a modulation signal (transmission signal) required by the light source 111 (or the light source in the flood light source module 14) to emit laser light, and the light source emits a light beam to a target object under the control of the modulation signal. For example, in one embodiment, the modulation signal is a continuous wave signal such as a sine wave signal, and the amplitude of the light source is modulated by the sine wave signal in the form of a sine wave in time sequence. In one embodiment, the modulation signal is a square wave signal or Pulse signal. The amplitude of the light source is modulated in time sequence under the modulation of the modulation signal to generate a square wave signal or a pulse signal for external emission.

In addition, the control and processing circuit 13 also provides the demodulated signal (collection signal) of each tap in each pixel of the TOF image sensor 121. The taps collect the electricity generated by the reflected light beam containing the target object under the control of the demodulated signal. signal. The electrical signal is related to the intensity of the reflected light beam. The control and processing circuit 13 then processes the electrical signal and calculates the intensity information reflecting the intensity of the reflected light beam to form a structured light pattern. Finally, based on the structured light pattern, it performs image matching calculations and triangulation. Method calculation and other calculations to obtain the depth image of the target object.

In some embodiments, the depth measuring device 10 further includes a flood light source module 14 for emitting flood light beams whose amplitude is modulated in time series. The control and processing circuit 13 can synchronously modulate and demodulate the flood light source module 14 and the TOF image sensor 121, so as to realize the traditional TOF ranging method to obtain the depth image of the target, that is, through the flood light beam from being emitted to being received It takes the time to calculate the depth value.

In some embodiments, the depth measurement device 10 may also include a drive circuit, a power supply, a color camera, an infrared camera, an IMU, and other devices, which are not shown in the figure. The combination with these devices can achieve richer functions, such as 3D. Texture modeling, infrared face recognition, SLAM and other functions. The depth measuring device 10 can be embedded in electronic products such as mobile phones, tablet computers, and computers.

Fig. 2 is a schematic diagram of the principle of the depth measuring device according to the first embodiment of the present invention. Under the control of the control and processing circuit 13, the light emitting module 11 emits a structured light beam 30 whose amplitude is sinusoidally modulated to the target object. In this embodiment, the structured light beam 30 is an irregular spot structured light, which is composed of multiple light spots. The light patterned light beam 201 with a spot structure formed in an irregular arrangement 202, and the amplitude of each light spot 202 is sinusoidally modulated in time sequence. Each pixel of the TOF image sensor in the acquisition module 12 includes at least 4 taps, which are respectively used to collect 4 optical signals and convert them into electrical signals C1, C2, C3 and C4 in a single frame period. The time of 4 acquisitions And the same interval.

The control and processing circuit 13 receives the electrical signals C1, C2, C3, and C4 to calculate the intensity information of the structured light beam. In one embodiment, the intensity information is calculated according to the following formula:

It is understandable that this calculation method is similar to the traditional structured light beam that is not amplitude modulated, that is, the intensity information of the optical signal is obtained by means of single-tap continuous integration within a single frame period, except that it is averaged here. After obtaining the intensity information of all pixels, the structured light pattern can be formed, and finally the structured light pattern is used for matching calculation to obtain the parallax and calculate the depth image according to the parallax.

When there is an ambient light signal, this beam intensity calculation method is the same as the traditional method, which is difficult to eliminate, resulting in a lower signal-to-noise ratio of the final structured light pattern. Therefore, in one embodiment, the intensity information will be calculated according to the following formula:

The above information essentially calculates the amplitude value of the amplitude modulation signal. The calculation of this amplitude value is due to the subtraction between taps and the square root, which can be regarded as the ambient light unchanged in a very small time range, so the subtraction is just fine It effectively removes the noise caused by ambient light, and the amplitude value also perfectly reflects the reflected light intensity information corresponding to the surface texture of the target object. It is understandable that the intensity information of the light beam can also be obtained by squaring the amplitude.

The above-mentioned structured light pattern acquisition scheme based on the 4-tap TOF image sensor and the sine wave modulated light emission signal is also applicable to other tapped TOF image sensors and other types of depth measurement devices with modulated light emission signals.

As shown in Fig. 3, it is a schematic diagram of the principle of the depth measuring device according to the second embodiment of the present invention. Under the control of the control and processing circuit 13, the light emitting module 11 emits a structured light beam 30 whose amplitude is modulated by a square wave or pulse to the target object. In this embodiment, the structured light beam is irregular spot structured light. That is, the spot structured light patterned light beam 201 formed by a plurality of light spots 202 in an irregular arrangement, and the amplitude of each light spot 202 is modulated by a square wave or pulse in time sequence. Each pixel of the TOF image sensor in the acquisition module 12 includes at least 3 taps, which are used to collect and convert the optical signal three times in a single frame period into electrical signals C1, C2, and C3. The trigger time of the first tap and The emission time of the light pulse is synchronized, and the trigger duration is the same as the pulse width. The second and third taps are triggered one after another at the end of the first tap, and the trigger duration is the same as the first tap.

Thus, the reflected structured light beam will be collected by the first, second, and third taps, and the control and processing circuit 13 receives the electrical signals C1, C2, and C3 to calculate the intensity information of the structured light beam. In one embodiment, the intensity information is calculated according to the following formula:

Similarly, although this method of using weighted average can calculate the value reflecting the intensity of the structured light beam, it cannot eliminate the noise problem caused by the background light like the traditional method. Therefore, in one embodiment, the intensity information will be calculated according to the following formula:

B=C ₁ +C ₂ -2C ₃ (4)

It is understandable that the above average can also be calculated to obtain the intensity information. Compared with formula (3), since the third tap collects the ambient light signal, the influence of ambient light can be effectively eliminated by subtraction, thereby improving the signal-to-noise ratio.

In the embodiments shown in Figures 2 and 3 above, the transmitter uses sine wave or pulse modulation, and the receiver uses 4-tap or 3-tap acquisition. When calculating the intensity information of the structured light pattern, a weighted average or an ambient light elimination factor can be used for calculation. It can be understood that the combination of similar modulation methods at the transmitting end, collection methods at the receiving end, and intensity information calculation methods are all included in the scope of the present invention.

Compared with the traditional structured light depth measurement method, the present invention adopts amplitude modulation at the transmitting end and multi-tap pixel acquisition at the receiving end, so that the method can have more functions than the traditional scheme, for example, it can realize the difficulty of the traditional scheme. Realized depth measurement method against ambient light interference.

Based on the depth measurement device of the foregoing embodiment, a fusion depth measurement solution combining structured light and TOF can also be realized. Fig. 4 is a schematic structural diagram of a depth measuring device integrating TOF and structured light solutions according to an embodiment of the present invention. The depth measuring device 40 includes a light emitting module 41, a collecting module 42, a flood light source module 44 and a control and processing circuit 43. Similar to the foregoing embodiments, the light emitting module 41, the acquisition module 42 and the control and processing circuit 43 can form a structured light depth measurement subunit, and the principle of acquiring depth images is the same as that of the embodiments shown in FIGS. 1 to 3 . In addition, the depth measurement device 40 consists of the flood light source module 44, the acquisition module 42 and the control and processing circuit 43 to form a TOF depth measurement sub-unit. Similarly, the light source in the flood light source module 44 is also controlled and processed. The circuit 43 performs amplitude modulation at a certain timing. For example, in one embodiment, in order to facilitate the design of the control and processing circuit 43, the light source in the flood light source module 44 and the light source in the light emitting module 41 are modulated in the same manner , Such as amplitude modulation intensity, modulation waveform and other parameters are the same.

The control and processing circuit 43 may also include a plurality of sub-circuits respectively corresponding to the structured light depth measurement sub-unit and the TOF depth measurement sub-unit. The difference between the TOF depth measurement subunit and the structured light depth measurement subunit is the depth calculation principle executed in the control and processing circuit 43. For the structured light depth measurement subunit, as described in the foregoing embodiments, the control and processing circuit 43 Process the electrical signal formed by the collection module 42 and calculate the intensity information reflecting the intensity of the reflected beam to form a structured light pattern, and then use the structured light triangulation method to calculate the depth value based on the structured light pattern; and for TOF depth measurement For the sub-unit, the control and processing circuit 43 processes the electrical signal formed by the acquisition module 42 to calculate the phase difference, and calculates the flight time of the reflected beam from the transmitting end to the receiving end based on the phase difference. The depth value of the target is further calculated based on the flight time. The control and processing circuit 43 can be highly integrated circuit components, such as FPGA, SOC, ASIC and other chips, which contain circuit modules that can perform structured light calculation and TOF calculation respectively. The present invention also provides a specific embodiment. This will be described in detail in the embodiment shown in FIG. 7.

Returning to FIG. 4, in this embodiment, the structured light depth measurement subunit and the TOF depth measurement subunit are respectively used to measure targets within ΔZ ₁ and ΔZ ₂ , and ΔZ ₁ ≠ΔZ ₂ . In one embodiment, ΔZ ₁ is a short distance, and ΔZ ₂ is a long distance, that is, the structured light depth measurement subunit measures a short distance target object, and the TOF depth measurement subunit measures a long distance target object. Because the accuracy of the structured light algorithm is high at short distances and will rapidly decrease or even impossible to measure with the increase of distance, the TOF algorithm has lower depth resolution when measuring at short distances, and has higher accuracy for long distances. Two subunits can be started for measurement at the same time, such as the form of switching between front and rear frames. After obtaining the structured light depth image and the TOF depth image respectively, the two depth images can be fused to obtain a large-scale, high-precision depth image. A weighted average based on confidence level can be used. For example, in one embodiment, the confidence is designed according to the relative accuracy. For structured light depth images, the greater the depth value, the lower the confidence, and for TOF depth images, the greater the depth value, the higher the confidence. The beneficial effect brought by the solution of this embodiment is that the depth measuring device 40 can achieve a larger measurement range while maintaining high measurement accuracy in a large range.

Fig. 5 is a schematic structural diagram of a depth measurement device integrating TOF and structured light solutions according to another embodiment of the present invention. The depth measurement device also includes a structured light depth measurement subunit and a TOF depth measurement subunit. It can be understood that the TOF depth measurement subunit can be composed of a flood light source module and a collection module, or can be combined with a structured light depth measurement subunit. The measurement subunit is the same composed of a light emitting module and an acquisition module. The difference is that the control and processing circuit processes the electrical signal in a different way. For the TOF depth measurement subunit, it calculates phase information. In this embodiment, the default TOF depth measurement subunit is composed of a flood light source module and an acquisition module, which can obtain the TOF depth value of each pixel of the acquisition module.

Since structured light depth calculation requires matching calculation, matching calculation is the most resource-consuming link in the entire depth calculation link and the link that has the greatest impact on accuracy. Matching calculation generally includes several steps such as initial depth value estimation, iterative optimization, and sub-pixel high-precision depth value calculation. The initial depth value estimation directly affects the efficiency and calculation accuracy of the subsequent steps. In this embodiment, the TOF depth value acquired by the TOF depth measurement subunit is used as the initial depth value for structured light depth calculation. As shown in FIG. 5, the TOF image sensor in the acquisition module includes a plurality of pixels 51, and the light emission mode The spot beams emitted by the group of light beams are reflected by the target object and then imaged to form a spot 52 on some pixels of the TOF image sensor. It is understandable that it is assumed that the spot 52 roughly occupies 2x2=4 pixels. In fact, the spot 52 can be of other sizes. There is no limitation here. First obtain the TOF depth value of each pixel 51 through the TOF depth measurement subunit (for the TOF depth measurement subunit composed of the light emitting module and the acquisition module, only the TOF depth value D1～D4 of the pixel corresponding to the light spot 52 can be obtained ), secondly, the structured light pattern containing each spot pattern is collected by the structured light depth measurement subunit, and the TOF depth value is used as the initial depth value d of the matching calculation when the structured light depth value is calculated finally.

Since the TOF depth value itself has high accuracy, it is used as the initial value of the structured light depth matching calculation, and then iterative calculation is carried out to realize the sub-pixel depth value estimation, which can achieve a higher precision depth than the structured light alone or TOF calculation. Calculation.

The embodiments shown in Figures 4 and 5 all involve the fusion of the structured light depth measurement subunit and the TOF depth measurement subunit. In actual operation, the two subunits will work in a certain time sequence. For example, Figure 6 is based on this A schematic diagram of the timing control of structured light and TOF according to an embodiment of the invention. In this embodiment, the structured light and TOF are switched to work at a certain frequency, that is, the depth measuring device works in structured light mode during periods T1, T3, T5, etc., In cycles T2, T4, T6, etc., it works in TOF mode. It is understandable that other timings can be used for control according to actual needs. For example, for the implementation shown in FIG. 5, the control can be performed in the manner of one frame of TOF and two frames of structured light.

Fig. 7 is a schematic diagram of a control and processing circuit architecture according to an embodiment of the present invention. In order to improve computing efficiency and reduce power consumption as much as possible, it is very effective to design some computing and control functions into dedicated chips, such as FPGA, ASIC or SOC chips. This embodiment provides a dedicated control and processing circuit chip architecture that can realize the control or calculation functions in the foregoing embodiments. The chip 70 is used to receive electrical signals from the TOF image sensor, and output a depth image after calculation by various internal modules. During the calculation process, it will also communicate with other devices, such as memory.

The chip 70 can implement multi-mode depth calculation, TOF depth calculation mode, structured light depth calculation mode, and fusion depth calculation mode.

(1) TOF depth calculation mode.

When TOF depth calculation is performed in the TOF depth calculation mode, the chip 70 receives electrical signals from the TOF image sensor, such as electrical signal values collected by each tap of the pixel in the embodiment shown in FIG. 2 or FIG. 3. The electrical signal will first enter the splitter 74-1. Under the control of the main controller (not shown in the figure), the splitter 74-1 selectively allows the electrical signal to enter the next calculation module according to the current mode. In the calculation mode, the electrical signal will enter the phase calculation module 71. The phase calculation module 71 performs phase calculation of the electrical signal to obtain the phase value, and there is a linear relationship between the phase value and the depth value. Therefore, in some embodiments, the phase calculation module can also directly calculate the depth value. Then the phase value is sent to the calibration module for calibration. Because TOF measurement is often interfered by noise, there is a certain error between the measured value and the actual value. Therefore, a calibration step will be adopted before actual use, such as within a certain measurement interval. The calibration board is set at intervals, and the actual depth value of the calibration board is known. Then, the calibration board at different distances is actually measured one by one to obtain the measured value corresponding to each distance. The relationship between the measured value and the actual value can be taken as The pre-calibration parameters are stored in the memory, and the calibration module will call the pre-calibration parameters from the memory to calibrate the current measured value during calibration. The pre-calibration parameter here can be a comparison table (index) between the actual value and the measured value. At this time, the calibration process of the calibration module 72 is actually a table look-up process; it can also be a mathematical method to model the error and pass Multiple measurements are taken in advance to calculate the unknown parameters in the model. The calibration process of the calibration module 72 is actually a process of calculating actual values based on the model and the measured values. The calibrated phase value/depth value will be sent to the first post-processing module 73. The first post-processing module 73 optimizes the phase value/depth value such as image enhancement, interpolation calculation, etc., such as hole filling, edge Optimization etc. The phase value/depth value processed by the first post-processing module 73 is output via the multiplexer 74-2.

(2) Structured light depth calculation mode.

When performing structured light depth calculation in the structured light depth calculation mode, the chip 70 receives electrical signals from the TOF image sensor, such as electrical signal values collected by each tap of the pixel in the embodiment shown in FIG. 2 or FIG. 3. The electrical signal will first enter the splitter 74-1. Under the control of the main controller (not shown in the figure), the splitter 74-1 selectively allows the electrical signal to enter the next calculation module according to the current mode. In the depth calculation mode, the electrical signal will enter the amplitude calculation module 75. The amplitude calculation module 75 performs phase calculation of the electrical signal to obtain intensity information reflecting the intensity of the beam, such as an amplitude value. Subsequently, the amplitude value is sent to the pre-processing module 76 for processing such as denoising, contrast enhancement, etc., and pre-processing tasks such as image distortion correction can also be performed on the amplitude value. The pre-processed amplitude value then enters the matching module 77 for matching calculation. During the matching calculation, the matching module 77 will recall the pre-stored reference image from the memory. In one embodiment, the matching module 77 uses zero mean normalization The minimized least square distance function estimates the pixel deviation between the image corresponding to the amplitude value and the reference image. According to the structured light triangulation method, there is a certain relationship between the pixel deviation value and the target depth value. Therefore, in some embodiments, the matching module 77 can also directly calculate the depth value. Of course, the depth value calculation can also be placed in the subsequent Other modules execute. The deviation value/depth value output by the matching module 77 is then sent to the second post-processing module 78 for image enhancement, interpolation calculation, etc. to optimize it, such as hole filling, edge optimization, etc. The deviation value/depth value processed by the second post-processing module 78 is output via the multiplexer 74-2.

(3) Fusion depth calculation mode.

When performing depth calculation in the fusion depth calculation mode, each module in the above two modes works, and the first post-processing module 73 outputs the TOF depth image, and the second post-processing module 78 outputs the structured light depth image, The TOF depth image and the structured light depth image are then sent to the fusion module 79 for fusion to output the final fusion depth image, and the fusion depth image is output through the multiplexer 74-2. For the specific fusion manner, refer to the embodiment shown in FIG. 4.

In some embodiments, after the TOF depth image output by the first post-processing module 73, it directly enters the matching module 77 as the initial value of the matching calculation. For the specific fusion scheme, refer to the embodiment shown in FIG. 5.

In some embodiments, the phase calculation module 71 and the amplitude calculation module 75 may be the same calculation module, which will synchronously calculate the phase and amplitude based on the electrical signal. The phase and amplitude will be directly sent to the next module for calculation, or be It is sent to the splitter 74-1, and the splitter 74-1 sends the phase and/or amplitude to the corresponding module for calculation according to the current mode.

In some embodiments, the above three modes can be performed synchronously, and the multiplexer 74-2 can synchronously or continuously output the depth images output by different modes in real time.

In some embodiments, the chip 70 may also include modules such as a bus, a main controller, an input/output interface, etc. For example, if the chip 70 is an integrated SOC chip, these modules will be included, and the splitter 74-1 receives the input through the input interface. Data, the data of the multiplexer 74-2 is output through the output interface, the input/output interface can be USB, MIPI, DVP and other forms of interface; the bus is responsible for the communication and transmission between the internal modules of the SOC and the external modules; the main controller It is responsible for resource allocation and function triggering among various internal modules.

Based on the depth measurement devices in the foregoing embodiments, the present application also provides the following depth measurement methods.

As shown in Figure 8, a distance measurement method in this application is as follows:

The light source emits a light beam whose amplitude is modulated in time sequence, and the light beam is received by the patterned optical element to form a structured light beam and emitted to the target object;

Using the pixel of the TOF image sensor containing at least one pixel to receive the structured light beam reflected by the target object and form an electrical signal;

The electric signal is received and the intensity information of the reflected structured light beam is calculated to form a structured light pattern, and the depth image of the target object is calculated by using the structured light pattern.

As shown in Figure 9, a distance measurement method in this application is as follows:

The light source of the light emitting module is used to emit a light beam whose amplitude is modulated in time sequence, and the light beam is received by the patterned optical element to form a structured light beam and emit it to the target object;

Using a flood light source module to emit a flood light beam whose amplitude is modulated in time sequence to the target object;

Using the pixel of the TOF image sensor containing at least one pixel to receive the structured light beam reflected by the target object and form a first electrical signal; and, receive the floodlight beam reflected by the target object and form a second electrical signal;

Control the timing of the light emitting module and the flood light source module, and respectively receive the first electrical signal formed by the structured light beam reflected by the target object according to the timing, and The second electrical signal formed by the floodlight beam reflected by the target object; the first electrical signal and the second electrical signal are respectively processed to obtain a depth image of the target object.

As shown in FIG. 10, in an embodiment, processing the electrical signal includes:

Calculating the intensity information of the structured light beam according to the first electrical signal formed by the structured light beam to form a structured light pattern, and calculating a first depth image of the target object by using the structured light pattern;

The phase difference is calculated according to the second electrical signal formed by the flood light beam, and the flight time required for the flood light beam from emission to reception is calculated based on the phase difference, and the flight time is calculated based on the flight time The depth value of the target object, using the depth value to obtain a second depth image of the target object;

Fusing the first depth image and the second depth image to obtain a depth image of the target object.

As shown in FIG. 11, processing the electrical signal in another embodiment includes:

The phase difference is calculated according to the second electrical signal formed by the flood light beam, and the flight time required for the flood light beam from emission to reception is calculated based on the phase difference, and each flight time is calculated based on the flight time TOF depth value of the pixel;

Calculating the intensity information of the structured light beam according to the first electrical signal formed by the structured light beam to form a structured light pattern;

The TOF depth value is used as the initial depth value calculated by the structured light pattern matching calculation, and the depth image of the target object is calculated by using the structured light pattern.

The present invention implements all or part of the processes in the above-mentioned embodiment methods, and can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed, the steps of the foregoing method embodiments can be realized. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file, or some intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory) , Random Access Memory (RAM, Random Access Memory), electrical carrier signal, telecommunications signal, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately added or deleted in accordance with the requirements of the legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to the legislation and patent practice, the computer-readable medium Does not include electrical carrier signals and telecommunication signals.

The beneficial effect achieved by the present invention is to provide a depth measurement device composed of a structured light emitting end based on amplitude timing modulation and a TOF image sensor, and propose a variety of depth image acquisition schemes based on the device to realize the current traditional High-precision, wide-range depth measurement that the solution is difficult to achieve At the same time, it also provides a deep computing chip architecture to realize the solution from the chip level, and finally realize the purpose of low power consumption and high efficiency computing.

The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be considered that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art to which the present invention belongs, without departing from the concept of the present invention, several equivalent substitutions or obvious modifications can be made, and the same performance or use should be regarded as belonging to the protection scope of the present invention.

Claims

The depth measuring device is characterized in that it comprises:

The light emitting module includes a light source and a patterned optical element, the light source is used to emit a light beam whose amplitude is modulated in time sequence, and the patterned optical element is used to receive the light beam and emit a structured light beam to a target object;

The TOF image sensor includes at least one pixel, and the pixel is used to receive the structured light beam reflected by the target object and form an electrical signal;

The control and processing circuit receives the electrical signal and calculates the intensity information of the reflected structured light beam to form a structured light pattern, and uses the structured light pattern to calculate a depth image of the target object.
The depth measuring device according to claim 1, wherein the structured light beam comprises an irregular spot pattern beam, a stripe pattern beam, and a two-dimensional coded patterned beam;

The amplitude of the beam corresponding to each spot in the irregular spot pattern beam is modulated in a continuous wave, square wave or pulse manner in time sequence.
The depth measuring device according to claim 1, wherein the pixel includes at least 3 taps, and the taps are used to separately collect within a single frame period generated by the structured light beam reflected by the target object. electric signal.
The depth measuring device according to claim 1, wherein the light emitting module emits a light beam whose amplitude is modulated by a sine wave in time sequence, and each pixel of the TOF image sensor includes at least 4 taps, which are respectively used in The optical signals are collected four times in a single frame period and converted into the electrical signals C1, C2, C3, and C4, and the time and interval of the four collections are the same;

The weighted average method is used to calculate the intensity B of the structured light beam, as follows:

Or, calculate the intensity B of the structured light beam by adding an ambient light elimination factor, as follows:
The depth measuring device according to claim 1, wherein the light emitting module emits a light beam whose amplitude is pulse-modulated in time sequence, and each pixel of the TOF image sensor includes at least 3 taps, which are respectively used for single The optical signal is collected three times in the frame period and converted into electrical signals C1, C2 and C3. The trigger time of the first tap is synchronized with the emission time of the optical pulse, and the trigger duration is the same as the pulse width. The second tap and the third tap Are triggered successively at the end of the first tap, and the trigger duration is the same as that of the first tap;

The weighted average method is used to calculate the intensity B of the structured light beam, as follows:

Or, calculate the intensity B of the structured light beam by adding an ambient light elimination factor, as follows:

B=C 1 +C 2 -2C 3 .
A distance measurement method, characterized in that it comprises:

The light source emits a light beam whose amplitude is modulated in time sequence, and the light beam is received by the patterned optical element to form a structured light beam and emitted to the target object;

Using the pixel of the TOF image sensor containing at least one pixel to receive the structured light beam reflected by the target object and form an electrical signal;

The electric signal is received and the intensity information of the reflected structured light beam is calculated to form a structured light pattern, and the depth image of the target object is calculated by using the structured light pattern.
7. The distance measurement method according to claim 6, wherein the structured light beam comprises an irregular spot pattern beam, a stripe pattern beam, and a two-dimensional coded patterned beam;

The amplitude of the beam corresponding to each spot in the irregular spot pattern beam is modulated in a continuous wave, square wave or pulse manner in time sequence.
7. The distance measuring method according to claim 6, wherein the pixel includes at least 3 taps, and the taps are used to separately collect within a single frame period generated by the structured light beam reflected by the target object. electric signal.
7. The distance measurement method according to claim 6, wherein the structured light beam is a beam whose amplitude is modulated by a sine wave in time series, and each pixel includes at least 4 taps, which are used to separate each in a single frame period. Collect 4 optical signals and convert them into the electrical signals C1, C2, C3, and C4, and the time and interval of the 4 collections are the same;

The weighted average method is used to calculate the intensity B of the structured light beam, as follows:

Or, calculate the intensity B of the structured light beam by adding an ambient light elimination factor, as follows:
The distance measurement method according to claim 6, wherein the structured light beam is a light beam whose amplitude is pulse-modulated in time series, and each of the pixels includes at least 3 taps, which are respectively used for collecting in a single frame period. The optical signal is three times and converted into electrical signals C1, C2 and C3. The trigger time of the first tap is synchronized with the emission time of the optical pulse, and the trigger duration is the same as the pulse width. The second tap and the third tap are in the first The taps are triggered one after another at the end, and the trigger duration is the same as the first tap;

The weighted average method is used to calculate the intensity B of the structured light beam, as follows:

Or, calculate the intensity B of the structured light beam by adding an ambient light elimination factor, as follows:

B=C 1 +C 2 -2C 3 .