KR20190115380A - Depth sensor control system - Google Patents

Abstract

The present invention relates to a depth sensor control system capable of accurately measuring a depth of an object by adjusting an amount of light received by the sensor. The depth sensor control system according to the present invention measures an IR intensity and frequency of the infrared rays reflected from the object at the light receiving part, and the accurate depth information of the object can be measured without additional equipment by adjusting the amount of light based on a created histogram based on the measurement at the light receiving part.

Description

Depth sensor control system

The present invention relates to a depth sensor control system capable of accurately measuring the depth of an object by automatically adjusting the amount of light received by the sensor.

In general, the automatic exposure control used in the image sensor is designed to maintain a target brightness for the continuous input image, and the exposure control is achieved through the control of the sensor gain and exposure time of the image sensor.

The automatic exposure control receives image data from the image sensor, processes it, and sends the accumulated time and gain to the sensor as it sees fit.

Here, the exposure includes the concept of charge integration time and gain. The charge accumulation time refers to the time taken until one pixel is reset and starts receiving light again to read out the accumulated charge amount. In addition, the gain represents the degree to which the charge generated in proportion to the accumulation time is amplified by an analog or digital method. In general, when there is sufficient lighting, exposure control usually only adjusts the charge accumulation time, leaving the gain at one.

However, in an environment with insufficient lighting, even if the exposure time is maximized, a sufficiently bright image cannot be obtained. Therefore, a bright image can be obtained by applying a gain greater than 1.

Hereinafter, with reference to the domestic registered patent (KR 10-1694621) and the domestic published patent (KR 10-2015-0037693), it will be described with respect to the image sensor for performing the conventional automatic exposure control.

1 and 2 are diagrams illustrating a conventional image sensor.

First, referring to FIG. 1, a brightness estimation apparatus 1 of a conventional image sensor according to a Korean registered patent KR 10-1694621 includes an image sensor 11, an automatic exposure control apparatus 12, and a brightness detector. 13, the LUT generating unit 14, and the histogram generating unit 15.

Here, the image sensor 11 includes a photographing sensor that outputs an RGB signal according to the intensity of light. The brightness detector 13 maps the sensor gain and exposure time output from the automatic exposure control device 12 with the sensor gain and exposure time stored in the LUT generator 14 to correspond to the corresponding brightness and the color gamut in an arbitrary color gamut. Calculate brightness information per pixel at.

However, this method has a limitation that is difficult to apply directly to the 3D sensor that can obtain the distance information with the object that is being studied recently.

Meanwhile, referring to FIG. 2, the laser device 2 according to the Korean Laid-Open Patent Application KR 10-2015-0037693 adjusts the light output by applying a laser light sensor.

Specifically, the laser device 2 measures the infrared rays emitted from the irradiation point of the laser light A formed on the processing material B using the sensor unit 32, or measures the temperature of the irradiation point, The control module 33 may correspondingly adjust the output in real time.

At this time, the control module 33 measures the temperature or the amount of light at the laser irradiation point using the sensor unit 32 in order to maintain the appropriate amount of light incident on the sensor unit 32, based on the laser module 31 Control the driving signal applied. The laser light A output from the laser module 31 is output through the light guide part 22 and the irradiation part 23.

However, in the case of the device for adjusting the light output, there is a problem that the manufacturing cost for the measuring device increases because a separate sensor for measuring the temperature and the light amount of the irradiation point is required.

In addition, in the conventional apparatus for adjusting the light output, since it takes a relatively long time to measure the temperature of the irradiation point, there is a problem that the time for adjusting the amount of light is increased.

An object of the present invention is to provide a depth sensor control system that automatically adjusts the amount of light for accurate depth measurement of an object.

In addition, an object of the present invention is to provide a depth sensor control system that can reduce the cost required to control the amount of light by adjusting the amount of incident light by using the infrared intensity measurement function built into the depth sensor.

In addition, an object of the present invention is to provide a depth sensor control system capable of ensuring a fast response time for light quantity adjustment for accurately measuring depth information of an object.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention, which are not mentioned above, can be understood by the following description, and more clearly by the embodiments of the present invention. Also, it will be readily appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

Depth sensor control system according to the present invention, by including a sensor control unit for controlling the exposure time of the light receiving unit and the output of the light emitting unit, it is possible to automatically adjust the appropriate amount of light for measuring the depth of the target object.

In addition, the depth sensor control system according to the present invention, by measuring the intensity and frequency of the infrared (IR intensity) reflected from the object in the light receiving unit, and adjusts the amount of light based on the histogram created based on this, it is accurate without additional equipment The depth information of the object can be measured.

In addition, the depth sensor control system according to the present invention, by controlling the exposure time of the light receiving unit and the output of the light emitting unit by using a histogram generated based on the intensity of the infrared ray measured by the light receiving unit, it is possible to achieve a fast response time for adjusting the amount of light of the depth sensor It can be secured.

Depth sensor control system according to the present invention, by automatically adjusting the amount of light for accurate depth measurement of the object, it is possible to prevent the phenomenon that the depth is not measured for some areas by the reflectance of the external light source or the object. Through this, the depth of the target object can be accurately measured by the depth sensor, and the reliability of the depth sensor can be improved.

In addition, the depth sensor control system according to the present invention, by adjusting the amount of incident light using the measurement function of the infrared intensity built into the depth sensor, to measure the exact depth of the object of interest without adding a component for the amount of light control can do. Through this, the manufacturing cost of the depth sensor can be reduced, thereby improving the profit of the manufacturer.

In addition, the depth sensor control system according to the present invention, by controlling the exposure time of the light receiving unit and the output of the light emitting unit by using a histogram created based on the intensity of the infrared ray measured by the light receiving unit, the amount of light for accurately measuring the depth information of the object Fast response time can be secured. Through this, the overall reaction speed of the system using the depth sensor can be improved, and the user's satisfaction using the device can be improved.

In addition to the effects described above, the specific effects of the present invention will be described together with the following description of specifics for carrying out the invention.

1 and 2 are diagrams illustrating a conventional image sensor.
3 is a block diagram illustrating a depth sensor control system according to an exemplary embodiment of the present invention.
4 is a view for explaining a driving method of the depth sensor control system of FIG.
FIG. 5 is a graph for explaining a histogram used by the sensor controller of FIG. 3.
FIG. 6 is a flowchart for describing an operation of the depth sensor control system of FIG. 3.

The terms or words used in this specification and claims are not to be construed as limiting in their usual or dictionary meanings, and the inventors may appropriately define the concept of terms in order to best describe their invention. Based on the principle that it can, it should be interpreted as meaning and concept corresponding to the technical idea of the present invention. In addition, the embodiments described in the specification and the drawings shown in the drawings are only one of the most preferred embodiments of the present invention, and do not represent all of the technical spirit of the present invention, it is possible to replace them at the time of the present application It should be understood that there may be various equivalents and variations in the range.

In general, an apparatus for sensing distance information with a target object includes a 3D camera, a depth sensor, a motion capture sensor, a laser radar, and the like.

Here, the depth sensor uses a time of flight (TOF) method.

The TOF method is a method of measuring the optical flight time until the light reflected from the target object is received by the sensor after irradiating the light to the target object. In this way, the depth sensor measures the distance to the object by measuring the time when the light emitted from the light source is reflected back to the object.

Hereinafter, the depth sensor control system for adjusting the amount of light incident on the depth sensor will be described in detail with reference to FIGS. 3 to 6.

3 is a block diagram illustrating a depth sensor control system according to an exemplary embodiment of the present invention. 4 is a view for explaining a driving method of the depth sensor control system of FIG.

3 and 4, the depth sensor control system 1000 according to an exemplary embodiment of the present invention includes a light emitter 100, a light receiver 200, and a controller 300.

In detail, the light emitter 100 emits light onto the object TG.

In this case, the light emitter 100 may irradiate the object TG with light in an infrared ray (IR) or near infrared ray (Near Infrared Ray) region.

For reference, this is only one example, and the light emitting unit 100 may emit light of different wavelengths (for example, a laser, an ultra-high frequency, a radio frequency (RF) signal, and an ultraviolet ray). Hereinafter, the light emitting unit 100 that emits infrared (IR) light will be described as an example.

Intensity and wavelength of the irradiated light may be adjusted according to a driving voltage or a magnitude of power applied to the light emitting unit 100. The output Ps of the light emitting unit 100 is controlled by the sensor controller 310.

Light emitted from the light emitter 100 may be reflected by the surface of the object TG, for example, skin or clothes. Depending on the distance between the light emitter 100 and the object TG, a phase difference between light emitted from the light emitter 100 and light reflected from the object TG may occur.

The light receiver 200 senses light (eg, infrared light) that is emitted from the light emitter 100 and reflected from the object TG.

The light receiver 200 includes a lens 210, an optical shutter 220, and an image sensor 230. However, this is merely an example and the light receiving unit 200 may be implemented by omitting some of the components or adding additional components.

In detail, the lens 210 collects infrared rays reflected from the object TG.

The light shutter 220 may be positioned on a path through which the light reflected from the object TG travels, and may change the IR intensity of the infrared light by changing the exposure time Texp of the reflected light.

In addition, the optical shutter 220 may modulate the waveform of the light reflected from the object TG by varying the degree of transmission of the light reflected from the object TG.

The light emitted from the light emitter 100 may be modulated by applying a specific frequency, and the optical shutter 220 may be driven at the same frequency as the specific frequency. In this case, the shape in which the reflected light is modulated by the optical shutter 220 may vary according to the phase of the light incident on the optical shutter 220.

Referring to FIG. 4, FIG. 4 illustrates changes in intensity of light emitted from the light emitting unit 100 over time (ILIR), and reflecting IR profile reflected from the object TG: Hereinafter, a graph of the intensity change over time of the RFIR is shown. In addition, changes in the transmittance of the optical shutter 220 with time are also shown.

The light emitter 100 may sequentially emit light ILIR to the object TG. In this case, the plurality of lights ILIR output from the light emitter 100 may be irradiated to the object TG with an idle time, and may be irradiated with different phases.

For example, when four lights ILIR are irradiated to the object TG from the light emitter 100, phases of the irradiated lights ILIR may be 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively. have.

Subsequently, the reflected light RFIRs reflected from the object TG may be incident to the image sensor 230 through the lens 210 and the optical shutter 220 independently.

In this case, the transmittance of the optical shutter 220 may change with time. The transmittance of the optical shutter 220 may vary according to the level of the bias voltage applied to the optical shutter 220 in a specific wavelength region. As a result, the waveform may be modulated while the reflected light RFIR passes through the optical shutter 220.

The waveform of the modulated reflected light RFIR may be changed according to the phase of the reflected light RFIR and the change in transmittance with time of the optical shutter 220.

Subsequently, the image sensor 230 may extract a phase difference between the reflected lights RFIR and the irradiated lights ILIR by photographing the reflected lights RFIR modulated by the optical shutter 220.

At this time, the image sensor 230 senses the intensity and phase of the light collected by the lens 210 and passed through the optical shutter 220. The image sensor 230 may include a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD).

Subsequently, the controller 300 may calculate depth information of the object TG based on the intensity and phase of the light sensed by the image sensor 230.

The controller 300 includes a sensor controller 310 and a depth calculator 320.

In detail, the sensor controller 310 may determine the exposure time Texp of the light receiver 200 or the light emitter 100 based on the light intensity (ie, the light quantity) of the light reflected from the object TG. Adjust the output (Ps).

When the intensity of light received by the light receiver 200 is not within an appropriate range, depth information of the object TG may not be properly measured.

To correct this, the sensor controller 310 reduces the exposure time Texp of the light receiver 200 or the output Ps of the light emitter 100 when the amount of light received is excessive.

On the contrary, when the received light amount is insufficient, the sensor controller 310 increases the exposure time Texp of the light receiver 200 or the output Ps of the light emitter 100.

Detailed description thereof will be described later.

Meanwhile, the depth calculator 320 calculates the phase difference of the light reflected from the object TG and calculates depth information of each pixel of the object TG based on the calculated difference.

That is, when the depth of some regions of the object TG is not measured because an appropriate amount of light is not secured by the external light source or the reflectance of the object TG, the controller 300 may adjust the intensity of light so that accurate depth information may be measured. It can be adjusted automatically.

For reference, the depth sensor control system 1000 of the present invention may include a display unit 400 that can visually display the depth information of the object TG to the user. However, this is only one example, and the present invention is not limited thereto.

In addition, although not clearly shown in the drawings, the depth sensor control system 1000 may transmit an operation command to the controller 300 using an interface unit (not shown).

For example, the interface unit (not shown) may include a touch panel provided on the display unit 400, a microphone for receiving a user's voice command, a device for recognizing a user's gesture, and the like.

FIG. 5 is a graph for explaining a histogram used by the sensor controller of FIG. 3.

Referring to FIG. 5, the sensor controller 310 adjusts an exposure time Texp of the light receiver 200 or an output Ps of the light emitter 100 to generate a histogram generated based on the intensity of the received light. I use it.

In detail, the sensor controller 310 generates a histogram with respect to the intensity of the infrared light incident on the light receiver 200.

Here, the X axis of the histogram represents the IR intensity, and the Y axis represents the number of pixels. For reference, the categories of the X axis and the Y axis for the histogram may be implemented in various modifications.

In the histogram, a range R (hereinafter, appropriate range R) for a proper intensity of infrared rays for accurate depth measurement is set.

The appropriate range R of the histogram is designated as an area where the depth can be most stably obtained through experiments, and may be stored in advance in the memory of the controller 300 and used.

For example, the appropriate range R of the histogram may be set to a range of 75 to 180 degrees of infrared intensity, but the present invention is not limited thereto.

In order for the depth calculator 320 to accurately measure the depth of the object TG, a ratio of the intensity of light belonging to the appropriate range R in the histogram should be positioned within a predetermined reference ratio range.

In this case, the reference ratio range may be set based on a look-up table that is experimentally generated and stored in advance. The look-up table may be previously stored in the memory of the controller 300 and used by the sensor controller 310.

When the ratio between the intensity of light belonging to the appropriate range R and the intensity of the entire light is not within the reference ratio range, the sensor controller 310 adjusts the exposure time Texp of the light receiver 200.

If the ratio of the light intensity belonging to the appropriate range (R) is greater than the upper limit of the reference ratio range, the exposure time (Texp) of the light receiving unit 200 is reduced, and if the ratio is less than the lower limit of the reference ratio range, the exposure time of the light receiving unit 200 Increase Texp.

For example, assuming that the reference ratio range is 70 percent to 80 percent, when the ratio of the light intensity within the appropriate range R is 60 percent, the sensor controller 310 may expose the exposure time Texp of the light receiver 200. Increase On the contrary, when the ratio of the light intensity within the appropriate range R is 90 percent, the sensor controller 310 reduces the exposure time Texp of the light receiver 200.

For reference, in another embodiment of the present invention, only a specific reference value may exist in the reference ratio range.

In this case, the sensor controller 310 adjusts the exposure time Texp of the light receiver 200 when the ratio of the intensity of light belonging to the appropriate range R and the intensity of the total light is equal to or less than the reference ratio range. .

For example, when the reference ratio range is 75 percent and the ratio of the light intensity within the appropriate range R is 70 percent, the sensor controller 310 increases the exposure time Texp of the light receiver 200.

On the other hand, the sensor controller 310 may adjust the exposure time Texp of the light receiver 200 only within a range of an appropriate exposure time.

Here, the appropriate exposure time range means a range of exposure time that can ensure a proper response speed and depth accuracy in the depth sensor control system 1000.

That is, the exposure time Texp of the light receiver 200 may be adjusted only between the upper limit threshold value T1 and the lower limit threshold value T2 of the appropriate exposure time range.

If the exposure time Texp of the light receiving unit 200 is adjusted to be located at the upper limit threshold T1 or the lower limit threshold T2, the sensor controller 310 may replace the exposure time Texp. Adjust the output Ps of 100).

Hereinafter, a method of adjusting the output Ps of the light emitting unit 100 in place of the exposure time Texp in order to secure an appropriate amount of light in the sensor controller 310 will be described in detail.

FIG. 6 is a flowchart for describing an operation of the depth sensor control system of FIG. 3.

Referring to FIG. 6, in operation of the depth sensor control system according to an exemplary embodiment of the present invention, the light emitter 100 emits infrared rays to the object TG (S110). The emitted infrared rays are reflected from the object TG and received by the light receiving unit 200.

Subsequently, the light receiver 200 detects infrared rays reflected from the object TG (S120). In this case, the IR intensity received by the light receiver 200 may vary depending on the reflectance of the external light source or the object TG.

Subsequently, the sensor controller 310 generates an infrared histogram based on the IR intensity received by the light receiver 200 (S130). At this time, the histogram (Histogram) is set to the appropriate range (R) of infrared rays for accurate depth measurement. Here, the appropriate range (R) of the histogram (Histogram) is designated as the area where the depth can be most stably through the experiment in advance, it may be stored and used in advance in the memory of the controller 300.

Subsequently, the sensor controller 310 calculates a ratio between the intensity of the infrared rays belonging to the appropriate range R and the intensity of the entire infrared rays (S140).

Subsequently, the sensor controller 310 derives a reference ratio range for securing reliability of the depth measurement based on a look-up table that has been experimentally generated and stored in advance.

Subsequently, the sensor controller 310 determines whether the calculated ratio is within a reference ratio range.

Subsequently, when the calculated ratio is within the reference ratio range, the depth calculator 320 calculates a depth for each pixel based on the data sensed by the light receiver 200 (S190).

On the other hand, when the calculated ratio is not included in the reference ratio range, the sensor controller 310 adjusts the exposure time Texp of the light receiver 200 (S160).

Specifically, the sensor controller 310 decreases the exposure time Texp of the light receiving unit 200 when the ratio of the intensity of infrared rays belonging to the appropriate range R is greater than the upper limit of the reference ratio range, and lower limit of the reference ratio range. If smaller, the exposure time Texp of the light receiver 200 is increased.

For reference, in another embodiment of the present invention, only a specific reference value may exist in the reference ratio range. In this case, the sensor controller 310 increases the exposure time Texp of the light receiver 200 when the ratio of the intensity of light belonging to the appropriate range R to the intensity of the total light is equal to or less than the reference ratio range. .

On the other hand, the sensor controller 310 may adjust the exposure time Texp of the light receiver 200 only within a range of an appropriate exposure time.

Subsequently, the sensor controller 310 determines whether the adjusted exposure time Texp is included in the appropriate exposure time range (S170). Here, the appropriate exposure time range means a range of exposure time that can ensure a proper response speed and depth accuracy in the depth sensor control system 1000.

That is, the exposure time Texp of the light receiver 200 may be adjusted only between the upper limit threshold value T1 and the lower limit threshold value T2 of the appropriate exposure time range.

If the adjusted exposure time Texp is not included in the appropriate exposure time range, the sensor controller 310 adjusts the output Ps of the light emitting unit 100 (S180).

In detail, when the adjusted exposure time Texp corresponds to an upper boundary value of the appropriate exposure time range, the sensor controller 310 increases the output of the light emitter 100. On the contrary, when the adjusted exposure time Texp corresponds to the lower limit value of the appropriate exposure time range, the sensor controller 310 reduces the output of the light emitter 100.

Subsequently, the sensor controller 310 repeatedly performs steps S110 to S170 described above.

Meanwhile, Even when the adjusted exposure time Texp is included in the appropriate exposure time range, the sensor controller 310 repeatedly performs steps S110 to S150 described above.

Through this, the sensor controller 310 may adjust the exposure time Texp of the light receiver 200 and the output Ps of the light emitter 100 until the appropriate infrared intensity incident on the light receiver 200 is obtained. The accuracy of the depth for the object TG calculated by the depth calculator 320 may be improved.

In addition, the depth sensor control system 1000 according to the present invention uses the histogram generated based on the intensity of the infrared rays measured by the light receiver 200, and the exposure time Texp of the light receiver 200 and the light emitter ( By controlling the output Ps of 100), it is possible to secure a fast response time with respect to the amount of light for accurately measuring depth information of the object.

Through this, the overall reaction speed of the depth sensor control system 1000 using the depth sensor may be improved, and the satisfaction of the user using the corresponding device may be improved.

In addition, the depth sensor control system 1000 according to the present invention adjusts the amount of incident light by using an infrared intensity measurement function built in the depth sensor, thereby accurately adjusting the target object without adding a component to control the amount of light. Depth can be measured. Through this, the manufacturing cost of the depth sensor can be reduced, and the profit of the manufacturer can be increased.

It is to be understood that the foregoing embodiments are illustrative in all respects and not restrictive, the scope of the invention being indicated by the claims that follow, rather than the foregoing detailed description. And it should be construed that the meaning and scope of the claims to be described later, as well as all changes and modifications derived from the equivalent concept are included in the scope of the present invention.

100: light emitting unit 200: light receiving unit
210: lens 220: optical shutter
230: image sensor 300: control unit
310: sensor control unit 320: depth calculation unit

Claims (9)

Light emitting unit for irradiating light to the object;
A light receiving unit configured to sense light reflected from the object;
A depth calculator configured to calculate a depth of a pixel based on the data sensed by the light receiver; And
A sensor controller configured to control an operation of the light receiver or the light emitter based on the data sensed by the light receiver,
The sensor control unit,
When the ratio of the proper range to the intensity of the light sensed by the light receiving unit is out of a predetermined reference ratio range, controlling the exposure time of the light receiving unit and the output of the light emitting unit
Depth sensor control system.
According to claim 1,
The sensor control unit,
A histogram is generated based on the intensity of light sensed by the light receiver;
Adjusting the exposure time of the light receiving unit based on the ratio of the light intensity corresponding to a predetermined appropriate range in the generated histogram
Depth sensor control system.
The method of claim 2,
The sensor control unit,
When the ratio is greater than the upper limit of the reference ratio range, the exposure time of the light receiving unit is reduced,
When the ratio is smaller than the lower limit of the reference ratio range, increasing the exposure time of the light receiving portion
Depth sensor control system.
The method of claim 2,
The sensor control unit,
When the adjusted exposure time is out of the appropriate exposure time range, adjusting the output of the light emitting unit
Depth sensor control system.
The method of claim 4, wherein
The sensor control unit,
When the adjusted exposure time corresponds to an upper boundary value of the appropriate exposure time range, the output of the light emitting unit is increased,
When the adjusted exposure time corresponds to the lower limit of the appropriate exposure time range, the output of the light emitting unit may be reduced.
Depth sensor control system.
The method of claim 4, wherein
The sensor control unit,
Repeatedly adjusting the exposure time of the light receiving unit or the output of the light emitting unit until the ratio of the light intensity corresponding to the appropriate range in the histogram is included in the reference ratio range.
Depth sensor control system.
According to claim 1,
The light emitting unit irradiates infrared rays to the object,
The light receiving unit measures an IR intensity and a frequency of infrared rays reflected from the object.
Depth sensor control system.
The method of claim 7, wherein
The sensor control unit,
The reference ratio range is determined using a pre-stored look-up table based on the intensity and frequency of the infrared rays.
Depth sensor control system.
According to claim 1,
The light receiving unit,
A lens for receiving light reflected from the object;
An image sensor for recognizing the light;
An optical shutter disposed between the lens and the image sensor and adjusting an amount of incident light;
Depth sensor control system.

Images