KR20120111092A - Image pick-up apparatus - Google Patents

Abstract

PURPOSE: A photographing device is provided to adaptively adjust a focus of a light receiving lens based on a depth map about subjects, thereby increasing a 3D realistic feeling. CONSTITUTION: A sensor module(100) generates depth data based on receiving light of subjects(50). An engine unit(300) generates a depth map about the subjects based on the depth data. The engine unit separates the subjects on the depth map. The engine unit generates a light receiving lens control signal based on the separated subjects. A motor unit(130) controls a focus of a light receiving lens(120) according to the control signal.

Description

Imaging device {IMAGE PICK-UP APPARATUS}

TECHNICAL FIELD The present invention relates to the field of imaging, and more particularly, to an imaging device capable of providing stereoscopic color data.

The image sensor is an apparatus for converting an optical signal including an image or distance or depth information into an electrical signal. In order to provide precise and accurate desired information, research on image sensor is underway. In particular, research and development of 3D image sensor that provides distance information as well as image information has been actively conducted recently. .

Accordingly, it is an object of the present invention to provide an imaging device capable of adaptively adjusting the focal length.

Another object of the present invention is to provide an imaging device that can enhance three-dimensional realism.

In order to achieve the above object, an imaging device according to an embodiment of the present invention includes a light receiving lens, a sensor module, an engine unit, and a motor unit. The sensor module generates depth data including distance information about the subjects based on the received light from the plurality of subjects. The engine unit generates a depth map for the subjects based on the depth data, separates the subjects on the depth map, and generates a control signal for controlling the light receiving lens based on the separated subjects. The motor unit controls the focus of the light receiving lens according to the control signal. The sensor module generates color data for the subjects based on visible light reflected from the subjects collected through the focus-adjusted light receiving lens, and provides the generated color data to the engine unit.

In an embodiment, the sensor module may include a depth sensor for generating the depth data and a color sensor for generating the color data.

The engine unit may process the depth data to generate a depth image and the depth map of the subjects, and separate the subjects based on the depth map, and based on the separated subjects. And a separation and control unit for generating the control signal for controlling the light receiving lens, and a second image signal processor for processing the color data to generate a color image of the subjects.

The image pickup device may further include an application for generating a stereo image of the subjects based on the depth image and the color image.

In example embodiments, the second image signal processor may process each of the subjects according to a distance between each of the subjects and the light receiving lens whose focus is adjusted.

In order to achieve the above object, the imaging apparatus according to the exemplary embodiment includes a light receiving lens, a sensor module, and an engine unit. The sensor module may be configured to determine depth data including distance information of the subjects based on the reflected light from the plurality of subjects through the light receiving lens and the visible light from the plurality of subjects. Generate color data. The engine unit generates a depth map for the subjects based on the depth data, and performs an image blurring process on the color data based on the depth map.

In example embodiments, the sensor module may include a depth / color sensor configured to simultaneously provide the depth data and the color data.

In example embodiments, the engine unit may process the depth data to generate a depth image of the subjects and the depth map, and separate the subjects based on the depth map, and separate the subjects. A separation unit for providing separation data relating to the data, a second image signal processor for processing the color data to generate a color image for the subjects, and performing an image blurring process for the color image based on the separation data It may include a blur processing unit for providing a blurred color image.

In an embodiment, the imaging device may further include an application for providing a stereo image for the subjects based on the depth image and the blurred color image.

In example embodiments, the light receiving lens may have a depth of field covering all of the subjects.

Based on the depth map of the plurality of subjects, the focus of the light receiving lens can be adaptively adjusted to improve the three-dimensional reality, and the blurring process on the color image based on the depth map can be used to perform the three-dimensional reality. Can be improved.

1 is a block diagram illustrating an image capturing apparatus according to an exemplary embodiment of the present invention.
2 is a block diagram illustrating a configuration of a sensor unit of FIG. 1 according to an exemplary embodiment of the present invention.
3 is a block diagram illustrating a configuration of a depth sensor of FIG. 2 according to an exemplary embodiment of the present invention.
4 is a diagram illustrating an example of calculating a distance of a subject in the depth sensor of FIG. 3.
5 is a block diagram illustrating a configuration of a sensor unit of FIG. 1 according to another exemplary embodiment of the present disclosure.
6 is a block diagram illustrating a configuration of the engine unit of FIG. 1 according to an exemplary embodiment of the present invention.
7 illustrates the host / application of FIG. 1 in accordance with an embodiment of the present invention.
8 illustrates a depth map of a plurality of subjects.
9A to 9C illustrate a subject selected in a depth map.
10A to 10C illustrate a color image focused on a subject selected in each of FIGS. 9A to 9C.
11 is a block diagram showing an image pickup device according to another embodiment of the present invention.
12 is a block diagram illustrating a configuration of the engine unit of FIG. 11 according to an embodiment of the present invention.
13 illustrates the host / application of FIG. 11 in accordance with an embodiment of the present invention.
14 illustrates a color image of a plurality of subjects.
15 illustrates a depth map of a plurality of subjects.
16A to 16C illustrate a blurred color image when the blurring process is performed based on the selected subject.
17 is a flowchart illustrating an image processing method according to an embodiment of the present invention.
18 is a flowchart illustrating an image processing method according to another exemplary embodiment of the present invention.
19 is a block diagram illustrating an example in which an imaging device according to an embodiment of the present invention is applied to a computing system.
20 is a block diagram illustrating an example of an interface used in the computing system of FIG. 19.

For the embodiments of the invention disclosed herein, specific structural and functional descriptions are set forth for the purpose of describing an embodiment of the invention only, and it is to be understood that the embodiments of the invention may be practiced in various forms, The present invention should not be construed as limited to the embodiments described in Figs.

The present invention is capable of various modifications and various forms, and specific embodiments are illustrated in the drawings and described in detail in the text. It should be understood, however, that the invention is not intended to be limited to the particular forms disclosed, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. In describing the drawings, similar reference numerals are used for the components.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between. Other expressions describing the relationship between components, such as "between" and "immediately between," or "neighboring to," and "directly neighboring to" should be interpreted as well.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, the terms "comprise", "having", and the like are intended to specify the presence of stated features, integers, steps, operations, elements, components, or combinations thereof, , Steps, operations, components, parts, or combinations thereof, as a matter of principle.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

Hereinafter, with reference to the accompanying drawings, it will be described in detail a preferred embodiment of the present invention. The same reference numerals are used for the same constituent elements in the drawings and redundant explanations for the same constituent elements are omitted.

1 is a block diagram illustrating an image capturing apparatus according to an exemplary embodiment of the present invention.

Referring to FIG. 1, the imaging apparatus 10 includes a sensor module 100, a light receiving lens 120, a motor unit 130, and an engine unit 300. The imaging device 10 may further include an application 400. The sensor module 100 may include a light source unit 110 and a sensor unit 200. The light source unit 110 may include a light source 111 and a lens 112. According to an exemplary embodiment, the sensor unit 200 and the light source unit 110 may be implemented as separate devices, or at least some components of the light source unit 110 may be included in the sensor unit 200.

The light receiving lens 120 may collect incident light into a light receiving area of the sensor unit 200 (for example, distance pixels and / or color pixels included in the pixel array included in the sensor unit 200). The sensor unit 200 first generates depth data ZDTA including distance information about the plurality of subjects 50 based on the received light RX reflected from the plurality of subjects 50, and then generates the engine unit ( 300). The engine unit 300 generates a depth map including distance information about the subjects 50 based on the depth data ZDTA, separates the subjects on the depth map based on the generated depth map, The control signal CTRL for controlling the light receiving lens 120 is generated based on the separated subjects. That is, the engine unit 300 selects one of the plurality of subjects on the depth map, sets the selected subject as the focus region, and provides a control signal CTRL for focusing the light receiving lens 120 to the set focus region. Can be generated.

The motor unit 130 moves the light receiving lens 120 according to the control signal CTRL provided from the engine unit 300 so that the light receiving lens 120 is focused on the subject selected by the engine unit 300. Accordingly, the sensor module 120 collects the color data CDTA of the subjects 50 based on the visible light VL reflected from the subjects 50 focused through the focus-adjusted light receiving lens 120. The generated color data CDTA is provided to the engine unit 300.

The light source 111 of the light source unit 110 generates infrared or near infrared rays, and the lens 112 allows the infrared or near infrared rays to focus on the subjects 50.

The sensor module 200 may provide the engine unit 300 with data DATA1 including the depth data ZDTA and the color data CDTA based on the clock signal CLK. According to an embodiment, the sensor module 200 may interface with the engine unit 300 through a mobile industry processor interface (MIPI) and / or a camera serial interface (CSI).

The engine unit 300 may control the sensor module 100 and the motor unit 130, and the engine unit 300 may process data DATA1 received from the sensor unit 200. For example, the engine unit 300 may generate stereoscopic color data (stereo data) based on the data DATA1 received from the sensor unit 200. In another example, the engine unit 300 generates YUV data including a luminance component, a difference between the luminance component and a blue component, and a difference between the luminance component and a red component based on the RGB data included in the data DATA1. Or compressed data, for example, Joint Photography Experts Group (JPEG) data. The engine unit 300 may be connected to the host / application 400, and the engine unit 300 may provide data DATA2 to the host / application 400 based on the master clock MCLK. In addition, the engine unit 300 may interface with the host / application 400 through a serial peripheral interface (SPI) and / or an inter integrated circuit (I2C).

In the embodiment of FIG. 1, the light receiving lens 120 may have a relatively short depth of field. That is, the focus of the light receiving lens 120 may be focused only on any one of the subjects 50.

2 is a block diagram illustrating a configuration of a sensor unit of FIG. 1 according to an exemplary embodiment of the present invention.

Referring to FIG. 2, the sensor unit 200a may include a depth sensor 210 and a color sensor 220. The depth sensor 210 may generate depth data ZDTA for the subjects 50 based on the received light RX from the subjects 50, including a depth pixel array composed of a plurality of depth pixels. have. In addition, the color sensor 220 may include a color pixel array composed of a plurality of color pixels to generate color data CDTA for the subjects 50 based on the visible light VL from the subjects 50. Can be.

3 is a block diagram illustrating a configuration of a depth sensor of FIG. 2 according to an exemplary embodiment of the present invention.

3 illustrates a case in which the light source unit 110 is included in the depth sensor 210 for convenience of description.

Referring to FIG. 3, the depth sensor 210 includes a depth pixel array 211, an analog-to-digital conversion (ADC) unit 212, a row scan circuit 213, and a column scan circuit 214. ), The controller 215 and the light source module 110.

The depth pixel array 211 includes depth pixels in which the light TX transmitted from the light source module 110 is reflected by the subjects 50 and converts the received light RX into an electrical signal. . The depth pixels may provide information about the distance of the subjects 50 from the depth sensor 210 together with the black and white image information.

The ADC unit 212 may convert the analog signal output from the depth pixel array 211 into a digital signal to provide the depth data ZDTA. According to an embodiment, the ADC unit 212 performs a column ADC for converting analog signals in parallel using an analog-digital converter connected to each column line, or sequentially converts the analog signals using a single analog-digital converter. You can perform a single ADC that converts to.

According to an embodiment, the ADC unit 212 may include a correlated double sampling (CDS) unit for extracting an effective signal component. In one embodiment, the CDS unit may perform analog double sampling to extract the valid signal component based on a difference between an analog reset signal representing a reset component and an analog data signal representing a signal component. In another embodiment, the CDS unit converts the analog reset signal and the analog data signal into digital signals, and then performs digital double sampling to extract a difference between two digital signals as the effective signal component. can do. In another embodiment, the CDS unit may perform dual correlation double sampling that performs both the analog double sampling and the digital double sampling.

The row scan circuit 213 may receive the control signals from the controller 215 to control the row address and the row scan of the depth pixel array 211. The row scan circuit 213 may apply a signal for activating the row line to the depth pixel array 211 in order to select the row line among the row lines. In one embodiment, the row scanning circuit 213 may include a row decoder to select a row line in the depth pixel array 211 and a row driver to supply a signal to activate the selected row line.

The column scan circuit 214 may control the column address and the column scan of the depth pixel array 211 by receiving control signals from the controller 215. The column scan circuit 214 may output the digital output signal output from the ADC unit 212 to a digital signal processing circuit (not shown) or an external host (not shown). For example, the column scan circuit 214 may sequentially select a plurality of analog-to-digital converters in the ADC unit 212 by outputting a horizontal scan control signal to the ADC unit 212. In one embodiment, column scan circuitry 214 may include a column decoder to select one of the plurality of analog-to-digital converters and a column driver to direct the output of the selected analog-to-digital converter to a horizontal transmission line. On the other hand, the horizontal transmission line may have a bit width for outputting the digital output signal.

The controller 215 may control the ADC unit 212, the row scan circuit 213, the column scan circuit 214, and the light source unit 110. The controller 215 may supply control signals such as a clock signal and a timing control signal required for the operation of the ADC unit 212, the row scan circuit 213, the column scan circuit 214, and the light source unit 110. In one embodiment, the controller 215 may include a logic control circuit, a phase lock loop (PLL) circuit, a timing control circuit, a communication interface circuit, and the like. In another embodiment, the function of the controller 215 may be performed by the engine unit 300.

The light source unit 110 may output light having a predetermined wavelength (for example, infrared rays or near infrared rays). The light source 111 may be controlled by the controller 215 to output light TX whose intensity is periodically changed. For example, the intensity of the light TX may be controlled to have a shape such as a pulse wave, sine wave, cosine wave, etc. having continuous pulses. The light source 210 may be implemented as a light emitting diode (LED), a laser diode, or the like.

4 is a diagram illustrating an example of calculating a distance of a subject in the depth sensor of FIG. 3.

In FIG. 4, an example of calculating a distance based on the received light RX from one of the subjects 50 will be described for convenience of description.

3 and 4, the light TX emitted from the light source unit 110 may have a varying intensity periodically. For example, the intensity of the emitted light TX (ie, the number of photons per unit area) can be in the form of a sine wave.

The light TX emitted from the light source unit 110 is reflected by the subject 50 and is incident on the depth pixel array 211 as the received light RX. The depth pixel array 211 may periodically sample the received light RX. According to an embodiment, the depth pixel array 211 has two sampling points having a phase difference of 180 degrees for each period of the received light RX (that is, the period of the emitted light TX), each 90 degrees of phase difference. The received light RX may be sampled at four sampling points having or, or more sampling points. For example, the depth pixel array 211 is capable of sampling samples A0, A1, A2, of received light RX at phases of 90 degrees, 180 degrees, 270 degrees and 360 degrees of emitted light TX every cycle. A3) can be extracted.

The received light RX may have an offset B different from the offset of the light TX emitted from the light source unit 100 due to additional background light, noise, or the like. The offset B of the received light RX may be calculated as shown in [Equation 1].

[Equation 1]

Here, A0 represents the intensity of the received light RX sampled at the 90 degree phase of the emitted light TX, and A1 represents the intensity of the received light RX sampled at the 180 degree phase of the emitted light TX. Where A2 represents the intensity of the received light RX sampled in the 270 degree phase of the emitted light TX, and A3 represents the intensity of the received light RX sampled in the 360 degree phase of the emitted light TX. Indicates.

The received light RX may have an amplitude A that is smaller than an amplitude of the light TX emitted from the light source unit 110 by the light loss. The amplitude A of the received light RX may be calculated as shown in Equation 2 below.

&Quot; (2) "

Black and white image information about the subject 50 may be provided based on the amplitude A of the received light RX for each of the distance pixels included in the depth pixel array 211.

The received light RX is delayed by the phase difference φ corresponding to twice the distance of the subject 50 from the depth sensor 210 with respect to the emitted light TX. The phase difference φ of the received light RX with respect to the emitted light TX may be calculated as shown in [Equation 3].

&Quot; (3) "

The phase difference φ of the received light RX with respect to the emitted light TX corresponds to the time-of-flight (TOF) of the light. The distance of the subject 50 from the depth sensor 210 can be calculated using the equation “R = c * TOF / 2”, where R represents the distance of the subject 160 and c represents the speed of light. Can be. In addition, the distance of the subject 50 from the depth sensor 210 may be calculated using Equation 4 using the phase difference φ of the received light RX.

&Quot; (4) "

Here, f denotes the modulation frequency, that is, the frequency of emitted light TX (or received light RX).

4 illustrates an example using light TX modulated to have a sine wave shape, but according to an exemplary embodiment, the depth sensor 210 may use various types of modulated light TX. In addition, the depth sensor 210 may extract the distance information in various ways according to the waveform of the intensity of the light TX, the structure of the distance pixel, and the like.

Since the color sensor 220 of FIG. 2 may have substantially the same structure as the depth sensor 210 of FIG. 3, a detailed description of the structure of the color sensor 220 of FIG. 2 will be omitted. That is, the color sensor 220 of FIG. 2 may generate color data CDTA for the subjects 50 including the color pixel array, the ADC unit, the row scan circuit, the column scan circuit, and the controller.

5 is a block diagram illustrating a configuration of a sensor unit of FIG. 1 according to another exemplary embodiment of the present disclosure.

Referring to FIG. 5, the sensor unit 200b includes a depth / color pixel array 230 in which a plurality of color pixels and depth pixels are arranged, color pixel selection circuits 250 and 270, and depth pixel selection circuits 240 and 290. ), A color pixel converter 260, and a depth pixel converter 280. The color pixel selection circuits 250 and 270 and the color pixel converter 260 control color pixels in the pixel array 230 to provide color data CDTA, and the depth pixel selection circuits 240 and 290 and depth pixel converters. 280 controls depth pixels in the pixel array 230 to provide depth information ZDTA.

As such, in the sensor unit 220b, components for controlling the color pixels and components for controlling the distance pixels are separately provided to provide the color data CDTA and the depth data ZDTA of the image. Can be.

Although not shown in FIG. 5, a control circuit such as FIG. 4 is included to include the color pixel selection circuits 250 and 270, the depth pixel selection circuits 240 and 290, the color pixel converter 260 and the depth pixel converter 280. Can be controlled.

6 is a block diagram illustrating a configuration of the engine unit of FIG. 1 according to an exemplary embodiment of the present invention.

In FIG. 6, the light receiving lens 120 and the motor unit 130 are shown together for convenience of description.

Referring to FIG. 6, the engine unit 300 may include a first image signal processor 310, a separation and control unit 320, and a second image signal processor 330.

The first image signal processor 310 processes the depth data ZDTA to generate a depth image ZIMG and a depth map DM of the subjects 50. The depth map DM may include depth information about the subjects 50, and the depth image ZIMG may be a black and white image including distance information about the subjects 50. The depth image ZIMG may be provided to the application 400, and the depth map DM may be provided to the separation and control unit 320. The separation and control unit 320 separates the subjects 50 on the depth map DM based on the depth map DM, and provides a control signal CTRL for controlling the light receiving lens 120 based on the separated subject. Create The control signal CTRL is provided to the motor unit 130, and the motor unit 130 moves the position of the light receiving lens 120 according to the control signal CTRL so that the light receiving lens 120 is separated from the control unit 320. Focus on the selected subject. Based on the visible light VL from the subjects 50 through the light receiving lens 120 focused on the subject selected by the separation and control unit 320, the sensor unit 200 receives the color data CDTA as the second image signal. To the processor 330. The second image signal processor 330 processes the color data CDTA to generate a color image CIMG.

That is, in the imaging apparatus 10 according to an exemplary embodiment, the depth map DM is first generated based on the distance information of the subjects 50, and the subject to be focused is selected based on the depth map DM. After the selection, the light receiving lens 120 may be moved to focus on the selected object, and then each of the subjects 50 may be color image processed according to the distance between each of the subjects 50 and the light receiving lens 120. That is, the subjects selected by the separation and control unit 320 among the subjects 50 and focused by the light receiving lens 120 may be processed with a large amount of calculation, and the remaining subjects may be processed with a smaller amount of calculation.

7 illustrates the host / application of FIG. 1 in accordance with an embodiment of the present invention.

Referring to FIG. 7, the host / application 400 may generate a stereo image SIMG, that is, a 3D color image by synthesizing the color image CIMG and the depth image ZIMG. That is, the host / application 400 synthesizes a black-and-white depth image ZIMG including distance information about the subjects 50 and a two-dimensional color image focused and processed on one of the plurality of subjects 50. It can provide a realistic three-dimensional color image (stereo image).

8 illustrates a depth map of a plurality of subjects, FIGS. 9A to 9C show a subject selected in the depth map, and FIGS. 10A to 10C show a color image focused on the selected subject in each of FIGS. 9A to 9C. .

Hereinafter, operations of the imaging apparatus 10 according to an exemplary embodiment of the present invention will be described in detail with reference to FIGS. 1 to 10C.

If the plurality of subjects 50 are located at different distances from the imaging apparatus 10, the received light RX from the subjects 50 reaches the sensor unit 200 with respect to the same transmission light TX. The depth map DM as shown in FIG. 8 can be obtained according to the time difference. When the user selects the subject SO1 on the depth map DM as shown in FIG. 9A, the separation and control unit 320 controls the light receiving lens 120 to focus on the selected subject SO1. To the motor unit 130. The motor unit 130 moves the light receiving lens 120 according to the control signal CTRL (for example, toward the sensor unit 200) to adjust the focus of the light receiving lens 120 to the selected object SO1. The sensor unit 200 generates color data CDTA based on the visible light from the subjects 50 through the light receiving lens 120 focused on the selected subject SO1, and generates the second data of the engine unit 300. The image signal processor 330 may process the color data CDTA to provide a color image CIMG as shown in FIG. 10A.

For example, when the user selects the subject SO2 on the depth map DM as shown in FIG. 9B, the separation and control unit 320 controls to control the light receiving lens 120 to focus on the selected subject SO2. The signal CTRL is provided to the motor unit 130. The motor unit 130 moves the light receiving lens 120 according to the control signal CTRL (for example, toward the sensor unit 200 or the subject 50), and focuses the light receiving lens 120 on the selected object SO2. ) The sensor unit 200 generates color data CDTA based on the visible light from the subjects 50 through the light receiving lens 120 focused on the selected subject SO2, and generates a second data of the engine unit 300. The image signal processor 330 may process the color data CDTA to provide a color image CIMG as shown in FIG. 10B.

For example, when the user selects the subject SO3 on the depth map DM as shown in FIG. 9C, the separation and control unit 320 controls the light receiving lens 120 to focus on the selected subject SO3. The CTRL is provided to the motor unit 130. The motor unit 130 moves the light receiving lens 120 (for example, toward the subject 50) according to the control signal CTRL to adjust the focus of the light receiving lens 120 to the selected subject SO3. The sensor unit 200 generates color data CDTA based on the visible light from the subjects 50 through the light receiving lens 120 focused on the selected subject SO3, and generates a second data of the engine unit 300. The image signal processor 330 may process the color data CDTA to provide a color image CIMG as shown in FIG. 10C.

11 is a block diagram showing an image pickup device according to another embodiment of the present invention.

Referring to FIG. 11, the imaging device 20 includes a sensor module 500, a light receiving lens 520, and an engine unit 700. The imaging device 20 may further include an application 700. The sensor module 500 may include a light source unit 510 and a sensor unit 600. The light source unit 510 may include a light source 511 and a lens 512. According to an exemplary embodiment, the sensor unit 600 and the light source unit 510 may be implemented as separate devices, or at least some components of the light source unit 510 may be included in the sensor unit 600.

The light receiving lens 520 may collect incident light into a light receiving area of the sensor unit 600 (eg, distance pixels and / or color pixels included in the pixel array included in the sensor unit 600). The sensor unit 600 generates depth data ZDATA including distance information about the plurality of subjects 50 based on the received light RX reflected from the plurality of subjects 60, and the subjects ( The engine unit 700 provides the color data CDTA including the color information of the subjects 60 based on the visible light VL from the 60. The engine unit 800 generates a depth map including distance information about the subjects 60 based on the depth data ZDTA, and blurs an image on the color data CDTA based on the generated depth map. Ring processing can be performed.

The light source 511 of the light source unit 510 generates infrared or near infrared rays, and the lens 512 allows the infrared or near infrared rays to focus on the subjects 60.

The sensor module 500 may provide the engine unit 800 with data DATA1 including the depth data ZDTA and the color data CDTA based on the clock signal CLK. According to an embodiment, the sensor module 200 may interface with the engine unit 300 through a mobile industry processor interface (MIPI) and / or a camera serial interface (CSI).

The engine unit 700 may control the sensor module 500, and the engine unit 700 may process data DATA1 received from the sensor unit 600. For example, the engine unit 700 may generate stereoscopic color data (stereo data) based on the data DATA1 received from the sensor unit 700. In another example, the engine unit 700 generates YUV data including a luminance component, a difference between the luminance component and a blue component, and a difference between the luminance component and a red component based on the RGB data included in the data DATA1. Or compressed data, for example, Joint Photography Experts Group (JPEG) data. The engine unit 700 may be connected to the host / application 800, and the engine unit 300 may provide data DATA2 to the host / application 400 based on the master clock MCLK. In addition, the engine unit 700 may interface with the host / application 400 through a Serial Peripheral Interface (SPI) and / or an Inter Integrated Circuit (I2C).

In the embodiment of FIG. 11, the light receiving lens 520 may have a relatively long depth of field. That is, the focus of the light receiving lens 520 may cover all of the subjects 60.

Since the sensor unit 600 may have a structure substantially the same as that of the sensor unit 200a of FIG. 2 or the sensor unit 200b of FIG. 5, a detailed description of the configuration and operation of the sensor unit 600 will be omitted. . That is, the sensor unit 600 may be implemented in which a depth sensor having a depth pixel array including a plurality of depth pixels and a color sensor having a color pixel array including a plurality of color pixels are separated from each other. It may also be implemented with a depth / color sensor comprising a pixel array in which pixels and depth pixels are arranged.

12 is a block diagram illustrating a configuration of the engine unit of FIG. 11 according to an embodiment of the present invention.

Referring to FIG. 12, the engine unit 700 may include a first image signal processor 710, a separation unit 720, a second image signal processor 730, and an image blur processing unit 740. The first image signal processor 710 processes the depth data ZDTA to generate a depth image ZIMG and a depth map DM of the subjects 60. The depth image ZIMG may be provided to the application 800, and the depth map DM is provided to the separator 720. The separating unit 720 separates the subjects 60 (select one of the subjects) based on the depth map DM, and provides separation data SDTA about the separated subjects. The second image signal processor 730 processes the color data CDTA to generate a color image CIMG of the subjects 60. The color image CIMG is provided to the blur processing unit 740. The blur processing unit 740 blurs the remaining subjects on the basis of the subject selected by the separating unit 720 among the color images CIMG based on the separation data SDTA, and blurs the color image BCIMG. ) For example, the blur processing unit 740 maintains the selected subject in the color image CIMG and performs the blurring process on the remaining subjects at different blur levels based on the relative distance from the selected subject. Can be.

13 illustrates the host / application of FIG. 11 in accordance with an embodiment of the present invention.

Referring to FIG. 13, the host / application 800 synthesizes the blurred color image BCIMG and the depth image ZIMG to generate a stereo image SIMG, that is, a three-dimensional color image including a blurring effect. Can be. That is, the host / application 800 is based on the black and white depth image ZIMG including the distance information about the subjects 60 and the separated data SDTA based on the depth map DM based on the distance information. The selected subject may be maintained among the subjects 60, and the remaining subjects may be synthesized with a blurred color image BCIMG to provide a more realistic three-dimensional color image (stereo image).

14 illustrates a color image of a plurality of subjects, FIG. 15 illustrates a depth map of a plurality of subjects, and FIGS. 16A to 16C show blurring when a blurring process is performed based on a selected subject. Represents a color image. For convenience of description, when the subject is selected as shown in FIGS. 9A to 9C in the depth map of FIG. 15A, the blurred color image of FIGS. 16A to 16C is illustrated.

Hereinafter, operations of the imaging apparatus 20 according to another exemplary embodiment of the present invention will be described in detail with reference to FIGS. 11 to 16C.

If the plurality of subjects 60 are located at different distances from the imaging device 20, the received light RX from the subjects 60 reaches the sensor unit 600 with respect to the same transmission light TX. According to the difference in time, a depth map DM as shown in FIG. 15 may be obtained. In this case, since the light receiving lens 520 has a relatively long depth of field, even if the plurality of subjects 60 are located at different distances from the imaging device 20, The color image CIMG is as shown in FIG. 14. That is, each of the plurality of subjects 60 is in focus and clearly appears.

For example, when the object SO1 is selected on the depth map DM as shown in FIG. 9A, the separation unit 720 provides the separation data SDTA representing the selected object SO1 to the image blur processing unit 740. In addition, the image blur processing unit 740 performs a blurring process on the remaining subjects according to depth information of each of the remaining subjects except for the selected subject SO1 to perform the blurred color image BCIMG as illustrated in FIG. 16A. Create For example, when the object SO2 is selected as shown in FIG. 9B on the depth map DM, the separation unit 720 provides the separation data SDTA representing the selected object SO2 to the image blur processing unit 740. In addition, the image blur processing unit 740 performs a blurring process on the remaining subjects according to the depth information of each of the remaining subjects except for the selected subject SO2 to generate the blurred color image BCIMG as shown in FIG. 16B. Create For example, when the object SO3 is selected as shown in FIG. 9C on the depth map DM, the separation unit 720 provides the separation data SDTA representing the selected object SO3 to the image blur processing unit 740. In addition, the image blur processing unit 740 performs a blurring process on the remaining subjects according to depth information of each of the remaining subjects except for the selected subject SO3 to generate the blurred color image BCIMG as illustrated in FIG. 16B. Create

17 is a flowchart illustrating an image processing method according to an embodiment of the present invention.

1 to 10 and 17, the sensor module 100 of the imaging apparatus 10 generates depth data ZDTA including distance information about the subjects 50 (S110). The engine unit 300 generates a depth map DM based on the depth data ZDTA (S120). The subjects are separated on the depth map DM, and a control signal CTRL for controlling the light receiving lens 120 is generated (S130). The motor unit 130 controls the focus of the light receiving lens 120 according to the control signal CTRL to focus on the separated subject (S140). The sensor module 100 generates color data CDTA for the subjects 50 based on the visible light collected through the focus-controlled light receiving lens 120 (S150). In an embodiment, the light receiving lens 120 may have a relatively short depth of field. That is, the focus of the light receiving lens 120 may be focused only on any one of the subjects 50.

18 is a flowchart illustrating an image processing method according to another exemplary embodiment of the present invention.

11 through 16 and 18, the sensor module 500 generates depth data ZDTA including distance information about the subjects 60 (S210). The sensor module 500 generates color data CDTA including image information of the subjects 60 in operation S220. The engine unit 700 generates a depth map DM based on the depth data ZDTA (S230). The engine unit 700 separates the subjects from the depth map DM, and generates separation data SDTA indicating the separated subjects (S240). The engine unit 700 performs an image blurring process on the color data CDTA based on the separation data SDTA to generate a blurred color image BCIMG (S250). In an embodiment, the light receiving lens 520 may have a relatively long depth of field. That is, the focus of the light receiving lens 520 may cover all of the subjects 60.

19 is a block diagram illustrating an example in which an imaging device according to an embodiment of the present invention is applied to a computing system.

Referring to FIG. 19, the computing system 1000 may include a processor 1010, a memory device 1020, a storage device 1030, an input / output device 1040, a power supply 1050, and an imaging device 900. have. Although not shown in FIG. 33, the computing system 1000 may further include ports for communicating with a video card, a sound card, a memory card, a USB device, or the like, or communicating with other electronic devices. .

Processor 1010 may perform certain calculations or tasks. According to an embodiment, the processor 1010 may be a micro-processor, a central processing unit (CPU). The processor 1010 may communicate with the memory device 1020, the storage device 1030, and the input / output device 1040 through an address bus, a control bus, and a data bus. have. In accordance with an embodiment, the processor 1010 may also be coupled to an expansion bus, such as a Peripheral Component Interconnect (PCI) bus. The memory device 1020 may store data necessary for the operation of the computing system 1000. For example, the memory device 1020 may be embodied as DRAM, mobile DRAM, SRAM, PRAM, FRAM, RRAM, and / or MRAM. have. The storage device 1030 may include a solid state drive, a hard disk drive, a CD-ROM, and the like. The input / output device 1040 may include input means such as a keyboard, a keypad, a mouse, and the like, and output means such as a printer or a display. The power supply 1050 can supply the operating voltage required for operation of the electronic device 1000. [

The imaging device 900 may be connected to the processor 1010 through the buses or other communication links to perform communication. As described above, the imaging apparatus 900 may first generate a depth map based on the distance information on the subjects, and adjust the focus of the light receiving lens according to the depth map. Alternatively, the imaging apparatus 900 may improve the 3D reality by performing a blurring process on the color images of the subjects based on the depth map of the subjects. The imaging apparatus 900 may be integrated on one chip together with the processor 1010 or may be integrated on different chips, respectively.

The imaging device 900 may be implemented in various types of packages. For example, at least some components of the three-dimensional image sensor 900 may be packaged on packages (PoP), ball grid arrays (BGAs), chip scale packages (CSPs), plastic leaded chip carriers (PLCC), plastic dual in- Line Package (PDIP), Die in Waffle Pack, Die in Wafer Form, Chip On Board (COB), Ceramic Dual In-Line Package (CERDIP), Plastic Metric Quad Flat Pack (MQFP), Thin Quad Flatpack (TQFP), Small Outline (SOIC), Shrink Small Outline Package (SSOP), Thin Small Outline (TSOP), Thin Quad Flatpack (TQFP), System In Package (SIP), Multi Chip Package (MCP), Wafer-level Fabricated Package (WFP), It can be implemented using packages such as Wafer-Level Processed Stack Package (WSP).

Meanwhile, the computing system 1000 should be interpreted as all computing systems using the imaging device. For example, the computing system 1000 may include a digital camera, a mobile phone, a personal digital assistant (PDA), a portable multimedia player (PMP), a smart phone, and the like.

20 is a block diagram illustrating an example of an interface used in the computing system of FIG. 19.

Referring to FIG. 20, the computing system 1100 may be implemented as a data processing device capable of using or supporting a MIPI interface, and may include an application processor 1110, an imaging device 1140, a display 1150, and the like. have. The CSI host 1112 of the application processor 1110 may perform serial communication with the CSI device 1141 of the 3D image sensor 1140 through a camera serial interface (CSI). In one embodiment, the CSI host 1112 may include a deserializer (DES), and the CSI device 1141 may include a serializer (SER). The DSI host 1111 of the application processor 1110 may perform serial communication with the DSI device 1151 of the display 1150 through a display serial interface (DSI).

In one embodiment, the DSI host 1111 may include a serializer (SER), and the DSI device 1151 may include a deserializer (DES). Further, the computing system 1100 may further include a Radio Frequency (RF) chip 1160 capable of communicating with the application processor 1110. The PHY 1113 of the computing system 1100 and the PHY 1161 of the RF chip 1160 may perform data transmission / reception in accordance with Mobile Industry Processor Interface (MIPI) DigRF. In addition, the application processor 1110 may further include a DigRF MASTER 1114 for controlling data transmission and reception according to the MIPI DigRF of the PHY 1161.

The computing system 1100 may include a Global Positioning System (GPS) 1120, a storage 1170, a microphone 1180, a dynamic random access memory (DRAM) 1185, and a speaker 1190. Can be. In addition, the computing system 1100 utilizes an ultra wideband (UWB) 1210, a wireless local area network (WLAN) 1220, a worldwide interoperability for microwave access (WIMAX) 1230, and the like. Communication can be performed. However, the structure and interface of the computing system 1100 is one example and is not limited thereto.

The imaging device according to embodiments of the present invention can be used in any light sensing device to provide a stereoscopic color image. The present invention is also useful for face recognition security systems, computers, digital cameras, 3D cameras, mobile phones, PDAs, scanners, car navigation systems, video phones, surveillance systems, auto focus systems, tracking systems, motion detection systems, image stabilization systems, and the like. Can be used.

As described above, the present invention has been described with reference to a preferred embodiment of the present invention, but those skilled in the art may vary the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. It will be understood that modifications and changes can be made.

Claims (10)

Light receiving lens;
A sensor module configured to generate depth data including distance information about the subjects, based on the received light from a plurality of subjects;
An engine unit generating a depth map of the subjects based on the depth data, separating the subjects on the depth map, and generating a control signal for controlling the light receiving lens based on the separated subjects; And
A motor unit for controlling the focus of the light receiving lens according to the control signal,
And the sensor module generates color data of the subjects and provides the generated color data to the engine unit based on visible rays reflected from the subjects focused through the focus-controlled light receiving lens. The method of claim 1, wherein the sensor module
A depth sensor for generating the depth data; And
And a color sensor for generating the color data. The method of claim 1,
The engine unit,
A first image signal processor configured to process the depth data to generate a depth image and the depth map of the subjects;
A separation and control unit for separating the subjects based on the depth map and generating the control signal for controlling the light receiving lens based on the separated subjects; And
And a second image signal processor configured to process the color data to generate a color image of the subjects. 4. The imaging device of claim 3, further comprising an application for generating a stereo image for the subjects based on the depth image and the color image. The image pickup apparatus of claim 3, wherein the second image signal processor processes each of the subjects according to a distance between the light receiving lens and the subjects whose focus is adjusted. Light receiving lens;
Generating color data of the subjects through the light receiving lens based on depth data including distance information of the subjects based on reflected light from a plurality of subjects and visible light from the plurality of subjects. A sensor module; And
And an engine unit generating a depth map of the subjects based on the depth data and performing an image blurring process on the color data based on the depth map. The imaging device of claim 6, wherein the sensor module comprises a depth / color sensor that provides the depth data and the color data simultaneously. The method of claim 6, wherein the engine unit,
A first image signal processor configured to process the depth data to generate a depth image and the depth map of the subjects;
A separation unit that separates the subjects based on the depth map and provides separation data regarding the separated subjects;
A second image signal processor configured to process the color data to generate a color image of the subjects; And
And a blurring processing unit which performs an image blurring process on the color image based on the separated data to provide a blurred color image. 10. The imaging device of claim 8, further comprising an application for providing a stereo image for the subjects based on the depth image and the blurred color image. The image pickup device of claim 6, wherein the light receiving lens has a depth of field covering all of the subjects.

Images