KR102106080B1 - Apparatus and method for detecting depth map - Google Patents

Abstract

The apparatus for extracting depth information according to an embodiment of the present invention includes an optical output unit outputting IR (InfraRed) light, an optical input unit outputting light reflected from an object after being output from the optical output unit, and an agent including the object. After adjusting the angle of the light so that light is irradiated to the 1 region, the light control unit for adjusting the angle of the light so that the light is irradiated to the second region that is part of the first region, and sequentially input over time with respect to the second region It includes a control unit for estimating the motion in the second region using the light.

Description

Depth information extraction device and method {APPARATUS AND METHOD FOR DETECTING DEPTH MAP}

The present invention relates to depth information extraction, and more particularly, to an apparatus and method for extracting depth information using a TOF (Time of Flight) method.

Technology for obtaining a 3D image using a photographing apparatus has been developed. Depth map is required to acquire a 3D image. Depth information is information indicating a distance in space, and indicates perspective information of another point with respect to one point of a 2D image.

One of the methods of acquiring depth information is a method of projecting IR (Infrared) structured light onto an object, and extracting depth information by analyzing light reflected from the object. According to the IR structured light system, there is a problem that it is difficult to obtain a desired level of depth resolution for a moving object.

As a technology that replaces the IR structured light system, the TOF (Time of Flight) method is drawing attention. According to the TOF method, the distance from the object is calculated by measuring the flight time, that is, the time when the light is reflected by shooting.

Generally, a camera conforming to the TOF method irradiates light to the front of an object by adjusting the angle of light. The TOF camera has a problem of low light efficiency and high computational power.

The technical problem to be achieved by the present invention is to provide a depth information extraction apparatus and method for extracting depth information using a TOF method.

The apparatus for extracting depth information according to an embodiment of the present invention includes an optical output unit outputting IR (InfraRed) light, an optical input unit outputting light reflected from an object after being output from the optical output unit, and an agent including the object. After adjusting the angle of the light so that light is irradiated to the 1 region, the light control unit for adjusting the angle of the light so that the light is irradiated to the second region that is part of the first region, and sequentially input over time with respect to the second region It includes a control unit for estimating the motion in the second region using the light.

The first region is an entire region including the object, and the second region is extracted from the first region and may be a partial region including a predetermined region in the object.

The control unit may calculate a flight time for the light output from the light output unit to be reflected after being irradiated to the second region and input to the light input unit.

The control unit may include a first flight time calculated using light input at the first time, a second flight time calculated using light input before the first time, and light input after the second time. Motion within the second region may be estimated using the third flight time calculated using.

The controller may estimate motion in the second region using a relative difference between the first flight time, the second flight time, and the third flight time, and then correct it using an interpolation technique.

The light input unit includes a plurality of pixels, each of which includes a first receiving unit and a second receiving unit, and the controller uses the difference in the amount of light input to the first receiving unit and the second receiving unit to fly the flight. Time can be calculated.

The light control unit may include a MEMS (Micro Electo Mechanical System) and a MEMS control unit.

The control unit includes a timing control unit that controls the time of the light output unit, the light adjustment unit, and the light input unit, a conversion unit that converts electrical signals input through the light input unit into digital signals, and the second region. It may include a signal processing unit for calculating the flight time of the light sequentially input with respect to time, estimate the motion, and extract the depth information.

The method of extracting depth information according to an embodiment of the present invention includes irradiating IR (InfraRed) light to a first region including an object, irradiating IR light to a second region that is part of the first region, Then, motion in the second region is estimated using light sequentially input to the second region over time.

The estimating step includes: a first flight time for light input at a first time, a second flight time for light input before the first time, and a first flight time for light input after the second time. 3 calculating a flight time, and estimating motion in the second region using a relative difference between the first flight time, the second flight time, and the third flight time.

According to the embodiment of the present invention, it is possible to obtain a depth information extraction device having a small amount of computation and excellent depth resolution. Accordingly, power consumption of the depth information extraction device can be reduced, and the distance of the object can be accurately extracted.

1 is a block diagram of a depth information extraction system according to an embodiment of the present invention.
2 shows a structure of an optical input unit of a depth information extraction apparatus according to an embodiment of the present invention.
3 shows a depth information extraction principle of a depth information extraction apparatus according to an embodiment of the present invention.
4 shows whole and partial regions sequentially photographed over time.
5 to 6 are flowcharts and flowcharts illustrating a depth information extraction method of a depth information extraction apparatus according to an embodiment of the present invention.

The present invention can be applied to various changes and can have various embodiments, and specific embodiments will be illustrated and described in the drawings. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms including ordinal numbers such as second and first may be used to describe various components, but the components are not limited by the terms. The terms are used only for the purpose of distinguishing one component from other components. For example, the second component may be referred to as a first component without departing from the scope of the present invention, and similarly, the first component may also be referred to as a second component. The term and / or includes a combination of a plurality of related described items or any one of a plurality of related described items.

When an element is said to be "connected" or "connected" to another component, it is understood that other components may be directly connected to or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that no other component exists in the middle.

The terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "include" or "have" are intended to indicate the presence of features, numbers, steps, actions, components, parts or combinations thereof described herein, one or more other features. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person skilled in the art to which the present invention pertains. Terms such as those defined in a commonly used dictionary should be interpreted as having meanings consistent with meanings in the context of related technologies, and should not be interpreted as ideal or excessively formal meanings unless explicitly defined in the present application. Does not.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings, but the same or corresponding components are assigned the same reference numbers regardless of reference numerals, and overlapping descriptions thereof will be omitted.

1 shows a block diagram of a depth information extraction system according to an embodiment of the present invention, FIG. 2 shows a structure of an optical input unit of a depth information extraction device according to an embodiment of the present invention, and FIG. 3 shows the structure of the present invention. A depth information extraction principle of a depth information extraction apparatus according to an embodiment is illustrated.

Referring to FIG. 1, the depth information extraction system includes a depth information extraction device 100 and a PC 200.

The depth information extraction apparatus 100 includes a light output unit 110, a light adjustment unit 120, a light input unit 130, and a control unit 140.

The light output unit 110 outputs infrared (IR) light. The IR light may be, for example, light having a wavelength band of 800 nm or more. The light output unit 110 includes a light source 112 and a light conversion unit 114. The light source may include at least one laser diode or light emitting diode (LED) that projects infrared light. Also, the light conversion unit 114 may modulate light output from the light source 112. The light conversion unit 114 may, for example, pulse modulate or phase modulate the light output from the light source 112. Accordingly, the light output unit 110 may output by blinking the light source at a predetermined interval.

The light adjusting unit 120 adjusts the angle of light so that light is irradiated to the area including the object. To this end, the light adjustment unit 120 may include a micro electo mechanical system (MEMS) 122 and a MEMS control unit 124.

The light adjusting unit 120 may adjust the angle of light so that light is irradiated to the entire area including the object. For example, as shown in FIG. 4 (a), the angle of light may be adjusted so that light is irradiated to the entire area including the person. Accordingly, the light output from the light output unit 110 may scan the entire area in units of pixels or lines. In addition, the light adjusting unit 120 may adjust the angle of light so that light is irradiated to a partial area that is a part of the entire area. For example, as shown in FIG. 4 (b), the angle of light may be adjusted so that light is irradiated to a partial area including the hand among the entire areas. Accordingly, the light output from the light output unit 110 may scan only a partial area in units of pixels or lines.

Meanwhile, the light input unit 130 receives light reflected by an object after being output from the light output unit 110. The optical input unit 130 may convert the received light into an electrical signal. The optical input unit 130 may be an image sensor including a photo diode (PD) or a complementary metal-oxide semiconductor (CMOS). As shown in FIG. 2, the light input unit 130 may include a plurality of pixels 132 arranged. Each pixel may include an In Phase receiving unit 132-1 and an Out Phase receiving unit 132-2.

The control unit 140 controls the depth information extraction apparatus 100 as a whole, and extracts depth information. The control unit 140 may be implemented as a controller chip. The control unit 140 may include a timing control unit 142, a conversion unit 144, a signal processing unit 146 and an interface controller 148. The timing control unit 142 controls the time of the light output unit 110, the light adjustment unit 120, and the light input unit 130. For example, the timing control unit 142 may control a flashing cycle of the light output unit 110. The conversion unit 144 converts the electrical signal input through the optical input unit 130 into a digital signal.

Then, the signal processing unit 146 calculates the flight time of light sequentially input with respect to the partial region, estimates motion, and extracts depth information. At this time, the flight time of the light may be calculated using the difference in the amount of light input to the In Phase receiving unit 132-1 and the Out Phase receiving unit 132-2. That is, as shown in FIG. 3, the In Phase receiving unit 132-1 may be activated while the light source is turned on, and the Out Phase receiving unit 132-2 may be activated while the light source is turned off. As described above, when the In Phase receiving unit 132-1 and the Out Phase receiving unit 132-2 are activated with a time difference, a difference occurs in the amount of light received according to the flight time of light, that is, the distance to the object. . For example, when the object is directly in front of the depth information extraction device (that is, when the distance = 0), since the time it takes for the light to be reflected after the light is output from the light output unit 110 is 0, the flashing cycle of the light source Becomes the reception cycle of light as it is. Accordingly, only the In Phase receiving unit 132-1 receives light, and the Out Phase receiving unit 132-2 cannot receive light. As another example, when the object is located at a predetermined distance from the depth information extraction device, since light is output from the light output unit 110 and it takes time to reflect on the object, the flashing cycle of the light source is different from the light receiving cycle. I am born. Accordingly, a difference occurs in the amount of light received by the In Phase receiving unit 132-1 and the Out Phase receiving unit 132-2.

The interface controller 148 controls an interface with middleware such as the PC 200. For example, the interface controller 148 may transmit information on the light input through the light input unit 130 to the middleware such as the PC 200 after irradiating the entire area. Then, after receiving information on the partial region extracted by the middleware, such as the PC 200, from the middleware, such as the PC 200, it can be transmitted to the light control unit 120.

5 to 6 are flowcharts and flowcharts illustrating a depth information extraction method of a depth information extraction apparatus according to an embodiment of the present invention. Descriptions overlapping with FIGS. 1 to 3 are omitted.

Referring to FIG. 5, the light output unit 110 of the depth information extraction apparatus 100 outputs IR light (S500), and the output light is an entire area including an object by adjustment of the light control unit 120 Investigate (S502).

Then, the light reflected from the object part is input through the light input unit 130 (S504), and the control unit 140 converts the analog signal received from the light input unit 130 into a digital signal (S506). 200) (S508).

The PC 200 extracts a partial region within the entire region using the signal received from the depth information extraction apparatus 100 (S510). The partial area may be an area including an object of interest required to implement an application application. For example, if the application application changes the TV channel according to the gesture of the finger, if the entire area includes the whole body of a person, the partial area may include only the finger.

The PC 200 transmits the information on the extracted partial region to the depth information extraction apparatus 100 (S512).

Meanwhile, the light output unit 110 of the depth information extraction device 100 outputs IR light (S514), and the output light irradiates only a partial area extracted from the entire area by the control of the light control unit 120. (S516).

Then, the light reflected from the object is input through the light input unit 130 (S518), and the controller 140 converts the analog signal received from the light input unit 130 into a digital signal (S520) and processes the signal. Depth information is extracted (S522).

According to an embodiment of the present invention, the depth information extraction apparatus 100 may scan a partial area several times during a time period in which the entire area can be scanned once. Accordingly, the daughter, steps S514 to S520 are repeated several times, and in step S522, the accuracy of depth information can be increased by using the repeated results. This will be described in detail with reference to FIG. 6.

Referring to FIG. 6, the control unit 140 of the depth information extraction device calculates the flight time of light input through the light input unit 130 at time T1 for the partial region (S600), and the light input unit 130 at T2. The flight time of the light input through is calculated (S602), and the flight time of the light input through the optical input unit 130 in T3 is calculated (S604). Here, T1, T2, and T3 may have the same time interval, and steps S514 to S520 in FIG. 5 may be repeated in T1, T2, and T3, respectively. As illustrated in FIG. 3, the flight time of light may be calculated by a difference in light amount between an In Phase receiving unit and an Out Phase receiving unit in a pixel of the optical input unit.

Then, the controller 140 estimates motion of the partial region based on the flight time from time T1 to time T3 (S606), and extracts depth information according to the motion estimation result (S608). At this time, the motion estimation and depth information extraction process may be performed according to a Super Resolution (SR) Algorithm. That is, as shown in FIG. 4 (b), after photographing the partial regions from T1 to Tn, each flight time may be calculated to estimate the relative motion over time. Then, the estimated motion is corrected by an interpolation technique, then restored, and noise reduction can be performed.

As described above, when the partial region is detected from the entire region and depth information for the partial region is extracted, computational complexity can be reduced. In addition, since motion information is estimated using a relative difference of information sequentially input over time with respect to the partial region, high depth resolution can be obtained. In addition, since the partial area can be scanned several times during the time of scanning the entire area once, it is possible to reduce the time and computational complexity of depth information extraction.

Although described above with reference to preferred embodiments of the present invention, those skilled in the art variously modify and change the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. You can understand that you can.

100: depth information extraction device
110: light output
120: light control unit
130: optical input
140: control unit

Claims (11)

Optical output unit that outputs IR (InfraRed) light,
An optical input unit through which light reflected from an object is input after being output from the optical output unit,
After adjusting the angle of the light so that light is irradiated to the first region including the object, the light adjusting unit for adjusting the angle of the light to be irradiated to the second region that is part of the first region, and
And a control unit for estimating motion in the second region by using light sequentially input to the second region with time,
The first region is the entire region including the object, the second region is extracted from the first region, and is a partial region including a predetermined region in the object,
The control unit calculates a flight time for the light output from the light output unit to be reflected after being irradiated to the second region and input to the light input unit,
The control unit may include a first flight time calculated using light input at a first time, a second flight time calculated using light input before the first time, and a light input after the first time. Depth information extraction apparatus for estimating motion in the second region by using the relative difference between the third flight time calculated using. delete delete delete According to claim 1,
The control unit is an apparatus for extracting depth information that corrects an estimated motion using an interpolation technique. According to claim 1,
The optical input unit includes a plurality of pixels, each of which includes a first receiving unit and a second receiving unit,
The controller is a depth information extraction device for calculating the flight time using the difference in the amount of light input to the first receiving unit and the second receiving unit. According to claim 1,
The light control unit is MEMS (Micro Electo Mechanical System) and depth information extraction device comprising a MEMS control unit. According to claim 1,
The control unit,
Timing control unit for controlling the time of the light output unit, the light control unit and the light input unit,
Conversion unit for converting the electrical signal input through the optical input unit to a digital signal, and
A signal processing unit that calculates flight time of light sequentially input with respect to the second region, estimates motion, and extracts depth information
Depth information extraction device comprising a. In the depth information extraction method of the depth information extraction apparatus,
Irradiating IR (InfraRed) light to the first region including the object,
Adjusting the angle of the IR light to irradiate the IR light to a second area that is part of the first area, and
And estimating motion in the second area using light sequentially input to the second area over time,
The first region is an entire region including the object, the second region is extracted from the first region, and is a partial region including a predetermined region in the object,
The estimating step,
Calculating a first flight time for light input at a first time, a second flight time for light input before the first time and a third flight time for light input after the first time Step, and
Estimating motion in the second region using a relative difference between the first flight time, the second flight time, and the third flight time
Depth information extraction method, including. delete According to claim 1,
The IR information output from the light output unit is a depth information extracting device that scans the partial area several times during a time to scan the entire area once.

Images