TW201640882A - Control method of a depth camera - Google Patents

Abstract

The present invention provides a control method of a depth camera including: by a processor, controlling a first image sensing unit and a second image sensing unit to operate so as to generate a first image and a second image respectively; by the processor, generating a first depth map based on the first image and the second image; by the processor, detecting edge pixels of the first image or the second image so as to obtain a edge pixel number; by the processor, determining whether the edge pixel number is smaller than a first predetermining pixel number; upon determining that the edge pixel number is smaller than the first predetermining pixel number, by the processor, controlling a light source to emit light; by the processor, controlling the second image sensing unit to operate so as to generate a third image; by the processor, generating a second depth map based on the third image; by the processor, fusing the first depth map with the second depth map so as to generate a fusion depth map; and, by the processor, registering the fusion depth map and the first image or the second image so as to generate 3D point cloud data.

Description

Depth camera control method

The present invention relates to a method for controlling a depth camera, and more particularly to a method for controlling a depth camera capable of generating three-dimensional point cloud data.

An existing depth camera is a structured light depth camera that uses an infrared light source to emit infrared light having a specific pattern, and uses an infrared light sensor to sense infrared light reflected from the field of view back to the camera to generate an infrared light image. A depth map can be obtained by analyzing the infrared image. The advantage of the structured light depth camera is that the depth map has high precision. However, when the distance of the object in the field of view of the structured light depth camera is far, the operating power of the infrared light source needs to be increased, and the emitted infrared light can be reflected from the field of view. Back to the structured light depth camera, it has the disadvantage of high power consumption.

Another existing depth camera is a stereoscopic depth camera that uses two color sensors to generate color images of different viewing angles, and by analyzing the color images, a depth map can be obtained. Since the stereoscopic depth camera does not need to emit infrared light, the power consumption is lower than that of the structured light depth camera. However, when a large flat scene appears in the field of view, the number of edge pixels obtained based on the color image is insufficient, which makes it impossible to effectively distinguish The distance between the scene and the depth camera.

Accordingly, it is an object of the present invention to provide a control method for a depth camera that can improve the disadvantages of the aforementioned conventional depth cameras.

Therefore, the depth camera of the present invention includes a light source, a first image sensing unit, a second image sensing unit spaced apart from the first image sensing unit and having an overlapping field of view, and a second image sensing unit Electrically connected to the light source, the first image sensing unit, and the processor of the second image sensing unit, the first image sensing unit is capable of sensing light of a first wavelength range, the second image sense The measuring unit can sense the light of the first wavelength range and the light of a second wavelength range different from the first wavelength range, and the wavelength of the light emitted by the light source is at the second wavelength The control method of the depth camera includes: (A) the processor controls the first image sensing unit and the second image sensing unit to sense the first wavelength range, so that the first image sensing The unit generates a first image and causes the second image sensing unit to generate a second image; (B) the processor generates a first depth map according to the first image and the second image; (C) the processing First to the device Performing edge detection on the second image to obtain an edge pixel number; (D) the processor determining whether the edge pixel number is less than a first preset pixel number; (E) when the processor determines the edge pixel number The processor controls the light source to emit light; (F) the processor controls the second image sensing unit to sense the second wavelength range, so that the second image sensing unit generates a third image; (G) the processor generates a second depth map according to the third image; (H) the processor uses the second depth map and the first depth map Fusion to generate a fusion depth map; and (I) the processor to register the fusion depth map with the first image or the second image to generate a three-dimensional point cloud data.

In some implementations, the method for controlling the depth camera further includes: (J) when the processor determines that the number of edge pixels is not less than the first preset number of pixels, the processor is configured according to the first image Calculating a depth representative value of the plurality of disparity values of the second image; (K) the processor determining whether the depth representative value is greater than a first preset depth; (L) when the processor determines that the depth representative value is not greater than The processor determines whether the number of edge pixels is smaller than a second preset pixel number, and the second preset pixel number is greater than the first preset pixel number; (M) when the processor determines the The number of edge pixels is smaller than the number of the second preset pixels, the processor determines whether the depth representative value is smaller than a second preset depth, the second preset depth is smaller than the first preset depth; and (N) when the processing Determining that the depth representative value is not less than the second preset depth, the processor controls the light source to emit light, and then performs (F), (G), (H), and (I), wherein the first source of the light source is driven The current value of a driving current is positive with the depth representative value turn off.

In some implementations, the method for controlling the depth camera further includes: (O) when the processor determines that the depth representative value is less than the second predetermined depth, the processor controls the light source to emit light, and then executes ( F), (G), (H), and (I), wherein a current value of a second driving current for driving the light source is not greater than a minimum current value of the first driving current.

In some implementations, the method for controlling the depth camera further includes: (P) when the processor determines that the depth representative value is greater than the first The preset depth does not control the light source to illuminate, and the first depth map is registered with the first image or the second image to generate a three-dimensional point cloud data.

In some implementations, the method for controlling the depth camera further includes: (Q) when the processor determines that the number of edge pixels is not less than the second predetermined number of pixels, the processor does not control the light source to emit light, and The first depth map is registered with the first image or the second image to generate a three-dimensional point cloud data.

The effect of the present invention is to determine whether to generate the second depth map and the operating power of the light source by determining the number of edge pixels and the magnitude of the depth representative value, thereby avoiding high power consumption and maintaining high-precision depth discrimination.

1‧‧‧Light source

2‧‧‧First image sensing unit

3‧‧‧Second image sensing unit

4‧‧‧ processor

5‧‧‧ memory

S01~S16‧‧‧ Process steps

I ₁ ‧‧‧First drive current

I ₂ ‧‧‧second drive current

Other features and effects of the present invention will be apparent from the following description of the drawings, wherein: FIG. 1 is a schematic diagram showing a hardware connection relationship of a depth camera according to an embodiment of the control method of the depth camera of the present invention; 2A and 2B are a flow chart of this embodiment.

Referring to Figures 1 and 2, there is shown an embodiment of a method of controlling a depth camera of the present invention. The depth camera includes a light source 1, a first image sensing unit 2, a second image sensing unit 3 spaced apart from the first image sensing unit and having an overlapping field of view, a memory 5, and an electrical connection to the light source. 1, the first An image sensing unit 2, a second image sensing unit 3, and a processor 4 of the memory 5.

The first image sensing unit 2 can sense light of a first wavelength range. The second image sensing unit 3 can sense the light of the first wavelength range and the light of a second wavelength range, and the second wavelength range is different from the first wavelength range. The wavelength of the light emitted by the light source 1 is in the second wavelength range. In this embodiment, the light in the first wavelength range is visible light, the light in the second wavelength range is invisible light, and the light emitted by the light source 1 is infrared light. Specifically, the first image sensing unit 2 of the embodiment includes an RGB sensor capable of sensing light of a first wavelength range, and the second image sensing unit of the embodiment includes a first wavelength capable of sensing RGB-IR sensor for light in the range and in the second wavelength range. In another embodiment, the first image sensing unit 2 includes an RGB sensor capable of sensing light of a first wavelength range, and the second image sensing unit includes a light capable of sensing a first wavelength range. An RGB sensor and an IR sensor capable of sensing light in a second wavelength range.

The memory 5 stores a first preset pixel number, a second preset pixel number greater than the first preset pixel number, a first preset depth, and a second preset smaller than the first preset depth. depth.

The control method of the depth camera is first, as shown in step S01, the processor 4 controls the first image sensing unit 2 and the second image sensing unit 3 to sense the first wavelength range, so that the first image sensing unit 2 generates a first image and a second image sensing unit 3 to generate a second image. Next, as shown in step S02, the processor 4 is configured according to the first image and the first The second image produces a first depth map.

Then, as shown in step S03, the processor 4 performs edge detection on the first image or the second image to obtain an edge-pixel number. Next, as shown in step S04, the processor 4 determines whether the number of edge pixels is smaller than the first preset number of pixels. If yes, step S05 is performed, and if no, step S10 is performed.

As shown in step S05, when the processor 4 determines that the number of edge pixels is smaller than the first preset number of pixels, the processor 4 controls the light source 1 to emit light. Then, as shown in step S06, the processor 4 controls the second image sensing unit 3 to sense the second wavelength range, so that the second image sensing unit 3 generates a third image. Next, as shown in step S07, the processor 4 generates a second depth map according to the third image. In this embodiment, the light source 1 can emit a patterned infrared light having a predetermined pattern, which can also be referred to as structured light, because the scenes of different distances in the field of view of the depth camera are illuminated by the structured light. The predetermined pattern illuminated on the scene may produce different shape changes, and the processor 4 can compare the various patterns in the third image with the single or multiple reference patterns previously stored in the memory 5, thereby Obtain the second depth map.

Next, as shown in step S08, the processor 4 fuses the second depth map with the first depth map to generate a merged depth map. Next, as shown in step S09, the processor 4 registers the merged depth map with the first image or the second image to generate a 3D point cloud data. Since the depth camera performs steps when the number of edge pixels is smaller than the first preset number of pixels S05~S09, so that when the accuracy of the first depth map is insufficient, the depth camera generates a second depth map with high precision, and the second depth map is merged with the first depth map, thereby maintaining the depth camera The accuracy of depth sensing.

On the other hand, when the result of the determination in step S04 is NO, that is, when the processor 4 determines that the number of edge pixels is not smaller than the number of the first preset pixels, the depth camera then performs step S10. Step S10: The processor 4 calculates a depth representative value according to the plurality of disparity values of the first image and the second image. More specifically, the first image and the second image have the same disparity value, and the processor 4 can calculate a depth from each disparity value, for example, reversing the disparity values and multiplying the first image by the first image. The spacing between the sensing unit 2 and the second image sensing unit 3 (based on the optical centers of the first image sensing unit 2 and the second image sensing unit 3), and multiplied by the first image sensing unit 2 Or the focal length of the second image sensing unit 3 is divided by the pixel size of the first image or the second image to determine the depth. The processor 4 then calculates the depth representative value based on a plurality of depths obtained using the aforementioned method. In this embodiment, the depth representative value is an average of the depths, but not limited thereto, and the depth representative value may also be, for example, a median of the depths.

Next, as shown in step S11, the processor 4 determines whether the depth representative value is greater than the first preset depth. If yes, step S16 is performed, otherwise step S12 is performed.

As shown in step S16, when the processor 4 determines that the depth representative value is greater than the first preset depth, the processor 4 does not control the light source 1 to emit light, and matches the first depth map with the first image or the second image. Quasi to produce 3D point cloud data. When the depth representative value is greater than the first preset depth, the processor 4 directly registers the first depth map with the first image or the second image without generating a second depth map, thereby avoiding the field of view. When the scenes are mostly too far away, the light source 1 of the depth camera operates at high power, resulting in high energy consumption.

As shown in step S12, when the processor 4 determines that the depth representative value is not greater than the first preset depth, the processor 4 determines whether the number of edge pixels is smaller than the second preset number of pixels. If yes, step S13 is performed. If not, step S16 is performed, that is, when the processor 4 determines that the number of edge pixels is not less than the second preset number of pixels, the processor 4 does not control the light source 1 to emit light, and the first depth map is The first image or the second image is registered to generate three-dimensional point cloud data, whereby when the number of edge pixels is sufficient to make the first depth map have high precision, the processor 4 directly directly applies the first depth The map is registered with the first image or the second image without generating a second depth map, thereby saving power consumption required to generate the second depth map.

On the other hand, as shown in step S13, when the processor 4 determines that the number of edge pixels is smaller than the second preset number of pixels, the processor 4 determines whether the depth representative value is smaller than a second preset depth, and if so, performs steps. S15. If no, step S14 is performed.

As shown in step S14, when the processor 4 determines that the depth representative value is not less than the second preset depth, the processor 4 controls the light source 1 to emit light, and is generated by the processor 4 and used to drive a first driving current of the light source 1. The current value of I ₁ is positively correlated with the depth representative value. The positive correlation here means that the current value of the first driving current is larger when the depth representative value is larger, and the smaller the current value of the first driving current I ₁ is when the depth representative value is smaller. Steps S06, S07, S08, and S09 are then performed after step S14. Therefore, when the distance between the scene object and the depth camera is not too close or too close, and the accuracy of the first depth map is not high or low, the depth camera adjusts the light source 1 according to the distance between the scene and the depth camera. The power is operated to generate a second depth map, and the second depth map is fused with the first depth map, so that the depth camera depth sensing accuracy can be maintained without causing excessive power consumption.

On the other hand, when the processor 4 determines that the depth representative value is smaller than the second preset depth, the processor 4 controls the light source 1 to emit light, and is generated by the processor 4 and used to drive a second driving current I ₂ of the light source 1. current value is not greater than the first driving current to the minimum current value I _1. Steps S06, S07, S08, and S09 are then performed after step S14. Thereby, when the distance between the field of view and the depth camera is very close, and the accuracy of the first depth map is not high or low, the depth camera operates the light source 1 with the lowest operating power to generate a second depth map, and the The second depth map is fused with the first depth map to maintain the accuracy of the depth camera depth sensing without causing excessive power consumption.

For example, the first preset pixel number is 5000, the second preset pixel number is 20000, the first preset depth is 10000 mm, and the second preset depth is 500 mm. When the number of the edge pixels is 2000, since the number of the edge pixels is smaller than the number of the first preset pixels, the depth camera performs steps S05 and S06 after performing the determining step of step S04. S07, S08 and S09.

When the number of edge pixels is 12000 and the depth representative value is 1000000 mm, since the number of edge pixels is not smaller than the first preset pixel number, and the depth representative value is greater than the first preset depth, the depth camera is executed. Step S16 is performed after the determining step of steps S04 and S11.

When the number of the edge pixels is 50,000 and the depth is 700 mm, the number of the edge pixels is not less than the number of the first preset pixels and the second predetermined number of pixels, and the depth representative value is not greater than the first pre-predetermined value. The depth is set. Therefore, the depth camera performs step S16 after performing the determining steps of steps S04, S11, and S12.

When the number of edge pixels is 10000 and the depth represents 700 mm, the number of edge pixels is between the first preset pixel number and the second preset pixel number, and the depth representative value is between Between a preset depth and a second preset depth, therefore, the depth camera performs step S14 after performing the determining steps of steps S04, S11, S12, and S13, and then performs steps S05, S06, S07, S08, and S09. .

When the number of edge pixels is 8000 and the depth representative value is 300 mm, the number of edge pixels is between the first preset pixel number and the second preset pixel number, and the depth representative value is smaller than the first The second preset depth, therefore, the depth camera performs step S15 after performing the determining steps of steps S04, S11, S12, and S13, and then performs steps S05, S06, S07, S08, and S09.

In summary, the control method of the depth camera of the present invention is determined by Breaking the number of edge pixels and the depth representative value to determine whether to generate the second depth map and the operating power of the light source 1, thereby avoiding high power consumption and maintaining high-precision depth discrimination, so the invention can be achieved. The purpose.

However, the above is only the embodiment of the present invention, and the scope of the present invention is not limited thereto, that is, the simple equivalent changes and modifications made by the patent application scope and the patent specification of the present invention are still It is within the scope of the patent of the present invention.

S01~S16‧‧‧ Process steps

Claims (5)

A depth camera control method, the depth camera includes a light source, a first image sensing unit, a second image sensing unit spaced apart from the first image sensing unit and having an overlapping field of view, and an electrical connection The light source, the first image sensing unit, and the processor of the second image sensing unit, the first image sensing unit can sense light of a first wavelength range, and the second image sensing unit can Sensing the light of the first wavelength range and the light of a second wavelength range, wherein the second wavelength range is different from the first wavelength range, and the wavelength of the light emitted by the light source is within the second wavelength range, The control method of the depth camera includes: (A) the processor controls the first image sensing unit and the second image sensing unit to sense the first wavelength range, so that the first image sensing unit generates a a first image, and the second image sensing unit generates a second image; (B) the processor generates a first depth map according to the first image and the second image; (C) the processor First image or the The second image performs edge detection to obtain an edge pixel number; (D) the processor determines whether the edge pixel number is less than a first preset pixel number; (E) when the processor determines that the edge pixel number is smaller than the first a preset number of pixels, the processor controls the light source to emit light; (F) the processor controls the second image sensing unit for the first The second wavelength range is sensed, so that the second image sensing unit generates a third image; (G) the processor generates a second depth map according to the third image; (H) the processor uses the second depth The map is fused with the first depth map to generate a fused depth map; and (1) the processor registers the fused depth map with the first image or the second image to generate a three-dimensional point cloud data. The control method of the depth camera according to claim 1, after (D), further comprising: (J) when the processor determines that the number of edge pixels is not less than the first preset number of pixels, the processor according to the first Calculating a depth representative value of an image and a plurality of disparity values of the second image; (K) the processor determining whether the depth representative value is greater than a first preset depth; (L) when the processor determines the depth The processor determines whether the number of edge pixels is smaller than a second preset pixel number, and the second preset pixel number is greater than the first preset pixel number; (M) when the The processor determines that the number of edge pixels is smaller than the number of the second preset pixels, and the processor determines whether the depth representative value is smaller than a second preset depth, where the second preset depth is smaller than the first preset depth; and (N When the processor determines that the depth representative value is not less than the second a preset depth, the processor controls the light source to emit light, and then performs (F), (G), (H), and (I), wherein a current value of a first driving current that drives the light source and the depth representative value Positive correlation. The control method of the depth camera according to claim 2, after (M), further comprising: (O) when the processor determines that the depth representative value is smaller than the second preset depth, the processor controls the light source to emit light, And then performing (F), (G), (H), and (I), wherein a current value of a second driving current that drives the light source is not greater than a minimum current value of the first driving current. The control method of the depth camera according to claim 2, after (K), further comprising: (P) when the processor determines that the depth representative value is greater than the first preset depth, the processor does not control the light source to emit light And registering the first depth map with the first image or the second image to generate a three-dimensional point cloud data. The control method of the depth camera according to claim 2, after (L), further comprising: (Q) when the processor determines that the number of edge pixels is not less than the second preset number of pixels, the processor does not control the The light source emits light, and the first depth map is registered with the first image or the second image to generate a three-dimensional point cloud data.