KR20210034301A - Camera module - Google Patents

Abstract

According to an embodiment of the present invention, provided is a camera module which can increase power efficiency of the camera module. The camera module comprises: a light emitting unit outputting a light signal to an object; a light receiving unit condensing the light signal outputted from the light emitting unit and reflected on the object; a sensor unit receiving the light signal condensed by the light receiving unit; and a control unit controlling at least one of the light emitting unit and the sensor unit to shift at least one of an output phase of the light emitting unit and a reception phase of the sensor unit.

Description

Camera module {CAMERA MODULE}

The embodiment relates to a camera module.

3D content is applied in many fields such as education, manufacturing, and autonomous driving as well as games and culture, and depth map is required to acquire 3D content. The depth information is information indicating a distance in space, and indicates perspective information of another point with respect to one point of a 2D image.

As a method of acquiring depth information, a method of projecting infrared (IR) structured light onto an object, a method of using a stereo camera, a method of using a time of flight (TOF), and the like are used. According to the TOF method, the distance to the object is calculated by measuring the flight time, that is, the time that light is emitted and reflected. The biggest advantage of the ToF method is that it provides fast, real-time distance information for a three-dimensional space. In addition, users can obtain accurate distance information without applying a separate algorithm or performing hardware correction. In addition, accurate depth information can be obtained by measuring a very close subject or measuring a moving subject.

Accordingly, there is an attempt to use the TOF method for biometric authentication. For example, it is known that the shape of veins spreading on fingers and the like does not change throughout life from the time of the fetus and varies from person to person. Accordingly, a vein pattern can be identified using a camera device equipped with a TOF function. To this end, after photographing a finger, each finger may be detected by removing a background based on the color and shape of the finger, and a vein pattern of each finger may be extracted from the detected color information of each finger. That is, the average color of the finger, the color of the veins distributed on the finger, and the color of the wrinkles on the finger may be different from each other. For example, the color of veins distributed on the fingers may be weaker in red than the average color of the fingers, and the color of the wrinkles on the fingers may be darker than the average color of the fingers. Using these features, a value approximating a vein can be calculated for each pixel, and a vein pattern can be extracted using the calculated result. In addition, an individual can be identified by comparing the extracted vein pattern of each finger and pre-registered data.

However, in the case of a camera device equipped with a TOF function, since a light source means for outputting light is further provided, power consumption is large. Such a problem may appear large in a terminal device having a limited battery capacity, such as a shift terminal. Therefore, there is a need to develop a ToF camera device capable of increasing power efficiency.

The technical problem to be achieved by the present invention is to provide a camera module with high power efficiency.

A camera module according to an embodiment of the present invention includes a light emitting unit that outputs an optical signal to an object; A light-receiving unit that is output from the light-emitting unit and condenses the optical signal reflected on the object; A sensor unit for receiving an optical signal condensed by the light receiving unit; And a control unit for controlling at least one of the light emitting unit and the sensor unit to shift at least one of an output phase of the light emitting unit and a reception phase of the sensor unit.

The duty ratio of the optical signal may be set within a range greater than 0% and less than 25%.

The control unit controls the light emitting unit or the sensor unit according to a plurality of sequences, and the plurality of sequences include a first sequence corresponding to a reference phase and a second sequence shifted by a predetermined phase from the reference phase. I can.

Each of the plurality of sequences may correspond to different phases.

The exposure period of the optical signal corresponding to the first sequence may be smaller than the exposure period of the optical signal corresponding to the second sequence.

The controller may set the phase of the second sequence based on an image signal generated in response to the first sequence.

The controller may perform each of the plurality of sequences during an exposure period corresponding to one image frame.

The control unit may control the sensor unit to shift the phase of the demodulated signal for the optical signal for each pixel of the sensor unit based on the image signal generated in response to the first sequence.

It may further include an image processing unit for generating a depth image by using a plurality of image signals corresponding to the plurality of sequences.

According to an embodiment of the present invention, it is possible to increase the power efficiency of the camera module.

In addition, even if a small amount of power is used, accurate distance information for an object can be measured.

1 is a configuration diagram of a camera module according to an embodiment of the present invention.
2 is a configuration diagram of a light emitting unit according to an embodiment of the present invention.
3 is a view for explaining a light receiving unit according to an embodiment of the present invention.
4 is a view for explaining a sensor unit according to an embodiment of the present invention.
5 is a diagram illustrating a process of generating an electric signal according to an embodiment of the present invention.
6 is a diagram for describing a sub-frame image according to an embodiment of the present invention.
7 is a diagram for describing a depth image according to an embodiment of the present invention.
8 is a view for explaining a ToF IR image according to an embodiment of the present invention.
9 is a view for explaining an optical signal of a light emitting unit according to an embodiment of the present invention.
10 is a view showing a measurable phase region according to an optical signal of a light emitting unit according to an embodiment of the present invention.
11 is a diagram for explaining an output phase shift of a light emitting unit according to an exemplary embodiment of the present invention.
12 is a diagram for explaining a phase shift of a reception of a sensor unit according to an embodiment of the present invention.
13 is a diagram for describing a measurable area for each phase according to an embodiment of the present invention.
14 is a diagram illustrating an embodiment of setting a shift phase of a control unit according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

However, the technical idea of the present invention is not limited to some embodiments to be described, but may be implemented in various different forms, and within the scope of the technical idea of the present invention, one or more of the constituent elements may be selectively selected between the embodiments. It can be combined with and substituted for use.

In addition, terms (including technical and scientific terms) used in the embodiments of the present invention are generally understood by those of ordinary skill in the art, unless explicitly defined and described. It can be interpreted as a meaning, and terms generally used, such as terms defined in a dictionary, may be interpreted in consideration of the meaning in the context of the related technology.

In addition, terms used in the embodiments of the present invention are for describing the embodiments and are not intended to limit the present invention.

In the present specification, the singular form may also include the plural form unless otherwise specified in the phrase, and when described as "at least one (or more than one) of A and (and) B and C", it is combined with A, B, and C. It may contain one or more of all possible combinations.

In addition, terms such as first, second, A, B, (a), and (b) may be used in describing the constituent elements of the embodiment of the present invention.

These terms are only for distinguishing the constituent element from other constituent elements, and are not limited to the nature, order, or order of the constituent element by the term.

And, when a component is described as being'connected','coupled' or'connected' to another component, the component is not only directly connected, coupled, or connected to the other component, but also with the component. It may also include the case of being'connected','coupled' or'connected' due to another element between the other elements.

In addition, when it is described as being formed or disposed on the “top (top) or bottom (bottom)” of each component, the top (top) or bottom (bottom) is one as well as when the two components are in direct contact with each other. It also includes the case where the above other component is formed or disposed between the two components. In addition, when expressed as "upper (upper) or lower (lower)", the meaning of not only an upward direction but also a downward direction based on one component may be included.

1 is a configuration diagram of a camera module according to an embodiment of the present invention.

The camera module 100 according to an embodiment of the present invention may be referred to as a camera device, a Time of Flight (ToF) camera module, a ToF camera device, or the like.

The camera module 100 according to an embodiment of the present invention may be included in an optical device. Optical devices include mobile phones, mobile phones, smart phones, portable smart devices, digital cameras, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), and navigation. It may include any one of. However, the type of optical device is not limited thereto, and any device for photographing an image or photograph may be included in the optical device.

Referring to FIG. 1, a camera module 100 according to an embodiment of the present invention includes a light emitting unit 110, a light receiving unit 120, a sensor unit 130 and a control unit 140, and an image processing unit 150 and A tilt unit 160 may be further included.

The light emitting unit 110 may be a light emitting module, a light emitting unit, a light emitting assembly, or a light emitting device. The light emitting unit 110 may output an optical signal. The light-emitting unit 110 may generate an optical signal and then output it to an object, that is, irradiate it. In this case, the light emitting unit 110 may generate and output an optical signal in the form of a pulse wave or a continuous wave. The continuous wave may be in the form of a sinusoid wave or a square wave. In this specification, the optical signal output from the light emitting unit 110 may mean an optical signal incident on an object. The optical signal output from the light emitting unit 110 may be referred to as an output light or an output light signal based on the camera module 100. The light output from the light emitting unit 110 may be referred to as an incident light or an incident light signal based on an object.

The light emitting unit 110 may output, that is, irradiate light to the object during a predetermined exposure period (integration time). Here, the exposure period may mean one frame period, that is, one image frame period. When generating a plurality of frames, the set exposure period is repeated. For example, when the camera module 100 photographs an object at 20 FPS, the exposure period is 1/20 [sec]. In addition, when 100 frames are generated, the exposure period may be repeated 100 times.

The light emitting unit 110 may output a plurality of optical signals having different frequencies. The light emitting unit 110 may sequentially and repeatedly output a plurality of optical signals having different frequencies. Alternatively, the light emitting unit 110 may simultaneously output a plurality of optical signals having different frequencies.

The light emitting unit 110 may set the duty ratio of the optical signal within a preset range. According to an embodiment of the present invention, the duty ratio of the optical signal output from the light emitting unit 110 may be set within a range greater than 0% or less than 25%. For example, the duty ratio of the optical signal may be set to 10% or 20%. The duty ratio of the optical signal may be preset or may be set by the controller 140.

The light-receiving unit 120 may be a light-receiving module, a light-receiving unit, a light-receiving assembly, or a light-receiving device. The light-receiving unit 120 may be output from the light-emitting unit 110 and collect the light signal reflected from the object. The light-receiving unit 120 may be disposed parallel to the light-emitting unit 110. The light receiving unit 120 may be disposed next to the light emitting unit 110. The light receiving unit 120 may be disposed in the same direction as the light emitting unit 110. The light receiving unit 120 may include a filter for passing the optical signal reflected on the object.

In the present specification, the optical signal collected by the light receiving unit 120 may mean an optical signal that is reflected after the optical signal output from the light emitting unit 110 reaches the object. The optical signal collected by the light receiving unit 120 may be referred to as an input light or an input light signal based on the camera module 100. The light output from the light receiving unit 120 may be referred to as a reflected light or a reflected light signal based on an object.

The sensor unit 130 may sense an optical signal collected by the light receiving unit 120. The sensor unit 130 may be an image sensor that senses an optical signal. The sensor unit 130 may be mixed with a sensor, an image sensor, an image sensor unit, a ToF sensor, a ToF image sensor, a ToF image sensor unit, and the like.

The sensor unit 130 may detect light and generate an electrical signal. That is, the sensor unit 130 may generate an electric signal through the optical signal collected by the light receiving unit 120. The generated electrical signal may be in an analog form. The sensor unit 130 may generate an image signal based on the generated electrical signal and transmit the generated image signal to the image processing unit 150. In this case, the image signal may be an analog type of electric signal or a signal obtained by digitally converting an analog type of electric signal. When transmitting an analog-type electrical signal as an image signal, the image processing unit 150 may digitally convert the electrical signal through a device such as an analog-digital converter (ADC).

The sensor unit 130 may detect light having a wavelength corresponding to the wavelength of light output from the light emitting unit 110. For example, the sensor unit 130 may detect infrared rays. Alternatively, the sensor unit 130 may detect visible light.

The sensor unit 130 may be a Complementary Metal Oxide Semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor. In addition, the sensor unit 130 may include a ToF sensor that measures a distance using a time or phase difference after receiving an infrared optical signal reflected on the subject.

The controller 140 may control each component included in the camera module 100.

According to an embodiment of the present invention, the control unit 140 may control at least one of the light emitting unit 110 and the sensor unit 130. According to an embodiment of the present invention, the control unit 140 may control at least one of the light emitting unit and the sensor unit to shift at least one of the output phase of the light emitting unit and the reception phase of the sensor unit. For example, the control unit 140 may control the light emitting unit 110 to shift the output phase of the optical signal according to a plurality of sequences. As another example, the control unit 140 may control the sensor unit 130 to shift the reception phase of the optical signal according to a plurality of sequences. As another example, the controller 140 may control the light emitting unit 110 and the sensor unit 130 to shift the output phase of the optical signal and the reception phase of the optical signal according to a plurality of sequences.

In addition, the control unit 140 may control the sensing period of the optical signal collected by the light receiving unit 120 of the sensor unit 130 in conjunction with the exposure period of the light emitting unit 110.

In addition, the control unit 140 may control the tilt unit 160. For example, the controller 140 may control the tilt driving of the tilt unit 160 according to a predetermined rule.

The image processing unit 150 may receive an image signal from the sensor unit 130 and process the image signal (eg, digital conversion, interpolation, frame synthesis, etc.) to generate an image.

According to an embodiment, the image processing unit 150 may synthesize one frame (high resolution) by using a plurality of frames (low resolution). That is, the image processing unit 150 may synthesize a plurality of image frames corresponding to the image signal received from the sensor unit 130 and generate the synthesized result as a synthesized image. The composite image generated by the image processing unit 150 may have a higher resolution than a plurality of image frames corresponding to an image signal. That is, the image processing unit 150 may generate a high-resolution image through a super resolution (SR) technique.

The image processing unit 150 may include a processor that generates an image by processing an image signal. A plurality of processors may be implemented according to functions of the image processing unit 150, and some of the plurality of processors may be implemented by being combined with the sensor unit 130. For example, a processor that converts an analog-type electrical signal into a digital-type image signal may be implemented in combination with a sensor. As another example, a plurality of processors included in the image processing unit 150 may be implemented separately from the sensor unit 130.

The tilt unit 160 may tilt at least one of the filter and the lens so that the optical path of light passing through at least one of the filter and the lens is repeatedly shifted according to a predetermined rule. To this end, the tilt unit 160 may include a tilt driver and a tilt actuator.

The lens may be a variable lens capable of changing the optical path. The variable lens may be a variable focus lens. Also, the variable lens may be a lens whose focus is adjusted. The variable lens may be at least one of a liquid lens, a polymer lens, a liquid crystal lens, a VCM type, and an SMA type. The liquid lens may include a liquid lens including one liquid and a liquid lens including two liquids. A liquid lens containing one liquid may change the focus by adjusting a membrane disposed at a position corresponding to the liquid, and for example, the focus may be changed by pressing the membrane by electromagnetic force of a magnet and a coil. A liquid lens including two liquids may control an interface formed between the conductive liquid and the non-conductive liquid by using a voltage applied to the liquid lens including a conductive liquid and a non-conductive liquid. The polymer lens can change the focus of the polymer material through a driving unit such as a piezo. The liquid crystal lens can change the focus by controlling the liquid crystal by electromagnetic force. The VCM type can change the focus by adjusting the solid lens or the lens assembly including the solid lens through electromagnetic force between the magnet and the coil. The SMA type can change the focus by controlling a solid lens or a lens assembly including a solid lens using a shape memory alloy.

The tilt unit 160 is a filter so that the path of light passing through the filter after tilting is shifted by a unit larger than 0 pixels of the sensor unit 130 and smaller than 1 pixel based on the path of light passing through at least one of the filter and lens before tilt And at least one of the lenses may be tilted. The tilt unit 160 may tilt at least one of the filter and the lens so that the path of light passing through at least one of the filter and the lens is shifted at least once from a preset reference path.

Hereinafter, each configuration of the camera module 100 according to the embodiment of the present invention shown in FIG. 1 will be described in detail with reference to the drawings.

2 is a configuration diagram of a light emitting unit according to an embodiment of the present invention.

As previously described with reference to FIG. 1, the light emitting unit 110 may refer to a component that generates an optical signal and then outputs it to an object. In order to implement this function, the light-emitting unit 110 may include a light-emitting element 111 and an optical element 112, and may include an optical modulator 112.

First, the light emitting device 111 may refer to a device that receives electricity and generates light (light). Light generated by the light emitting device 111 may be infrared rays having a wavelength of 770 to 3000 nm. Alternatively, the light generated by the light emitting device 111 may be visible light having a wavelength of 380 to 770 nm.

The light emitting device 111 may include a light emitting diode (LED). In addition, the light emitting device 111 may include an organic light emitting diode (OLED) or a laser diode (LD).

The light emitting device 111 may be implemented in a form arranged according to a certain pattern. Accordingly, the light emitting device 111 may be configured in plural. The plurality of light emitting devices 111 may be arranged in rows and columns on the substrate. The plurality of light emitting devices 111 may be mounted on a substrate. The substrate may be a printed circuit board (PCB) on which a circuit pattern is formed. The substrate may be implemented as a flexible printed circuit board (FPCB) in order to secure a certain flexibility. In addition, the substrate may be implemented with any one of a resin-based printed circuit board, a metal core PCB, a ceramic PCB, and an FR-4 substrate. In addition, the plurality of light emitting devices 111 may be implemented in the form of a chip.

The optical modulator 112 may control blinking of the light emitting element 111 to control the light emitting element 111 to generate an optical signal in the form of a continuous wave or a pulse wave. The optical modulator 112 may control the light emitting element 111 to generate light in the form of a continuous wave or a pulse wave through frequency modulation or pulse modulation. For example, the light modulator 112 may control to generate pulsed or continuous light by repeating blinking (on/off) of the light emitting element 111 at regular time intervals. The predetermined time interval may be the frequency of the optical signal.

3 is a view for explaining a light receiving unit according to an embodiment of the present invention.

Referring to FIG. 3, the light receiving unit 120 includes a lens assembly 121 and a filter 125. The lens assembly 121 may include a lens 122, a lens barrel 123, and a lens holder 124.

The lens 122 may be composed of a plurality of sheets, or may be composed of one sheet. The lens 122 may include the variable lens described above. When the lens 122 is formed of a plurality of elements, each of the lenses may be aligned with respect to a central axis to form an optical system. Here, the central axis may be the same as the optical axis of the optical system.

The lens barrel 123 is coupled to the lens holder 124, and a space for accommodating a lens may be provided therein. The lens barrel 123 may be rotationally coupled to one or a plurality of lenses, but this is exemplary, and may be coupled in other ways, such as a method using an adhesive (for example, an adhesive resin such as epoxy). .

The lens holder 124 may be coupled to the lens barrel 123 to support the lens barrel 123, and may be coupled to the printed circuit board 126 on which the sensor 130 is mounted. Here, the sensor may correspond to the sensor unit 130 of FIG. 1. A space in which the filter 125 can be attached may be formed under the lens barrel 123 by the lens holder 124. A helical pattern may be formed on the outer circumferential surface of the lens holder 124, and similarly, the helical pattern may be rotationally coupled to the lens barrel 123 having a helical pattern formed on the outer circumferential surface. However, this is exemplary, and the lens holder 124 and the lens barrel 123 may be coupled through an adhesive, or the lens holder 124 and the lens barrel 123 may be integrally formed.

The lens holder 124 may be divided into an upper holder 124-1 coupled to the lens barrel 123 and a lower holder 124-2 coupled to the printed circuit board 126 on which the sensor 130 is mounted. , The upper holder 124-1 and the lower holder 124-2 may be integrally formed, formed in a structure separated from each other, and then fastened or combined, or may have a structure separated from each other and spaced apart from each other. In this case, the diameter of the upper holder 124-1 may be formed smaller than the diameter of the lower holder 124-2.

The filter 125 may be coupled to the lens holder 124. The filter 125 may be disposed between the lens assembly 121 and the sensor. The filter 125 may be disposed on the optical path between the object and the sensor. The filter 125 may filter light having a predetermined wavelength range. The filter 125 may pass light of a specific wavelength. That is, the filter 125 may block by reflecting or absorbing light other than a specific wavelength. The filter 125 may pass infrared rays and block light having a wavelength other than infrared rays. Alternatively, the filter 125 may pass visible light and block light having a wavelength other than visible light. Filter 125 is shiftable. The filter 125 may be integrally shifted with the lens holder 124. The filter 125 may be tilted. The filter 125 can be shifted to adjust the optical path. The filter 125 may change a path of light incident on the sensor unit 130 through a shift. The filter 125 may change a field of view (FOV) angle of incident light or a direction of the FOV.

Although not shown in FIG. 3, the image processing unit 150 or the like may be implemented in a printed circuit board. In addition, the light emitting unit 110 of FIG. 1 may be disposed on the side of the sensor 130 on the printed circuit board 126, or may be disposed outside the camera module 100, for example, on the side of the camera module 100. have.

The above example is only an example, and the light receiving unit 120 may be configured with another structure capable of condensing light incident on the camera module 100 and transmitting it to the sensor.

4 is a view for explaining a sensor unit according to an embodiment of the present invention.

As shown in FIG. 4, in the sensor unit 130 according to an embodiment of the present invention, a plurality of cell areas P1, P2, ... may be arranged in a grid shape. For example, as shown in FIG. 4, in the sensor unit 130 having a resolution of 320x240, 76,800 cell areas may be arranged in a grid form.

In addition, a certain interval L may be formed between each cell region, and a wire or the like for electrically connecting a plurality of cells may be disposed at the interval L. The width dL of the gap L may be very small compared to the width of the cell area.

The cell regions P1, P2, ... may refer to regions for converting an input light signal into electrical energy. That is, the cell regions P1, P2, ... may refer to a cell region in which a photodiode converting light into electrical energy is provided, or a cell region in which the provided photodiode operates.

According to an embodiment, two photodiodes may be provided in the plurality of cell regions P1, P2, .... Each of the cell regions P1, P2, ... has a first light receiving unit 132-1 including a first photodiode and a first transistor, and a second light receiving unit 132-including a second photodiode and a second transistor. 2) may be included.

The first light receiving unit 132-1 and the second light receiving unit 132-2 may receive an optical signal with a phase difference of 180 degrees from each other. That is, when the first photodiode is turned on to absorb the optical signal and then turned off, the second photodiode may be turned on to absorb the optical signal and then turned off. The first light receiving unit 132-1 may be referred to as an in phase receiving unit, and the second light receiving unit 132-2 may be referred to as an out phase receiving unit. In this way, when the first light receiving unit 132-1 and the second light receiving unit 132-2 are activated with a time difference, a difference occurs in the amount of light received according to the distance to the object. For example, when the object is in front of the camera module 100 (that is, when the distance is 0), the time it takes to be reflected from the object after the optical signal is output from the light emitting unit 110 is 0, so the light source blinks. The period becomes the light reception period as it is. Accordingly, only the first light receiving unit 132-1 receives the light, and the second light receiving unit 132-2 does not receive the light. As another example, when the object is located a predetermined distance away from the camera module 100, since it takes time to be reflected from the object after the light is output from the light emitting unit 110, the flashing period of the light source is different from the light reception period. I will. Accordingly, a difference occurs in the amount of light received by the first light receiving unit 132-1 and the second light receiving unit 132-2. That is, the distance of the object may be calculated by using the difference between the amount of light input to the first light receiving unit 132-1 and the second light receiving unit 132-2.

5 is a diagram illustrating a process of generating an electric signal according to an embodiment of the present invention.

As shown in FIG. 5, there may be _four reference signals, that is, demodulation signals, according to an embodiment of the present invention (C ₁ to C 4 ). Each of the demodulation signals C ₁ to C ₄ may have the same frequency as the output light (light output from the light emitting unit 110 ), that is, the incident light from the viewpoint of the object, but may have a phase difference of 90 degrees from each other. One of the four demodulated signals C ₁ may have the same phase as the output light. The phase of the input light (light received by the light receiving unit 120), that is, the reflected light from the viewpoint of the object, is delayed by a distance from which the output light enters the object and then returns. The sensor unit 130 mixes the input light and each demodulation signal, respectively. Then, the sensor unit 130 may generate an electric signal corresponding to the shaded portion of FIG. 3 for each demodulation signal. The electric signal generated for each demodulation signal may be transmitted to the image processing unit 150 as an image signal, or a digitally converted electrical signal may be transmitted to the image processing unit 150 as an image signal.

In another embodiment, when the output light is generated at a plurality of frequencies during the exposure time, the sensor absorbs the input light according to the plurality of frequencies. For example, _{it is assumed that output light is generated at frequencies f 1} and f ₂ , and that a plurality of demodulated signals has a phase difference of 90 degrees. Then, since incident light also has frequencies f ₁ and f ₂ , four electrical signals may be generated through input light having a frequency of f _{1 and four demodulation signals corresponding thereto.} In addition, four electrical signals may be generated through an input light having a frequency of f _{2 and four demodulated signals corresponding thereto.} Thus, a total of eight electrical signals can be generated.

6 is a diagram for describing a sub-frame image according to an embodiment of the present invention.

As described above, the electric signal may be generated corresponding to the phase of each of the four demodulated signals. Accordingly, as illustrated in FIG. 6, the image processing unit 150 may obtain sub-frame images corresponding to four phases. Here, the four phases may be 0°, 90°, 180°, and 270°, and the sub-frame image may be mixed with a phase image, a phase IR image, and the like.

In addition, the image processing unit 150 may generate a depth image based on a plurality of sub-frame images.

7 is a diagram for describing a depth image according to an embodiment of the present invention.

The depth image of FIG. 7 represents an image generated based on the sub-frame image of FIG. 4. The image processing unit 150 may generate a depth image using a plurality of sub-frame images, which may be implemented through Equations 1 and 2 below.

Here, Raq(x ₀ ) means a sub-frame image corresponding to a 0 degree phase. Raq(x ₉₀ ) means a sub-frame image corresponding to a 90 degree phase. Raq(x ₁₈₀ ) means a sub-frame image corresponding to a 180 degree phase. Raq(x ₂₇₀ ) means a sub-frame image corresponding to a 270 degree phase.

That is, the image processing unit 150 may calculate a phase difference between an optical signal output by the light emitting unit 110 and an optical signal input by the light receiving unit 120 for each pixel through Equation 1.

Here, f means the frequency of the optical signal. c stands for the speed of light.

That is, the image processing unit 150 may calculate the distance between the camera module 100 and the object for each pixel through Equation 2.

Meanwhile, the image processing unit 150 may generate a ToF IR image based on a plurality of sub-frame images.

8 is a view for explaining a ToF IR image according to an embodiment of the present invention.

FIG. 8 shows an amplitude image, which is a kind of ToF IR image generated through the four sub-frame images of FIG. 4.

In order to generate the amplitude image as shown in FIG. 8, the image processing unit 150 may use Equation 3 below.

As another example, the image processing unit 150 may generate an intensity image, which is a type of ToF IR image, using Equation 4 below. The intensity image may be mixed with a confidence image.

A ToF IR image such as an amplitude image or an intensity image may be a gray image.

9 is a view for explaining an optical signal of a light emitting unit according to an embodiment of the present invention.

The light emitting unit 110 may output an optical signal having a predetermined duty ratio. The light emitting unit according to the exemplary embodiment of the present invention may output an optical signal having a duty ratio in a range of greater than 0% and less than 25%.

9 shows light power of an optical signal according to a duty ratio of 25% and a duty ratio of 10%. As shown in FIG. 9, the smaller the duty ratio is, the shorter the time the light is output during one exposure period. The smaller the duty ratio is, the shorter the time the light is outputted, thus reducing power consumption. However, due to the shortening of the light output time, there may be a section in which the distance measurement is impossible.

10 is a view showing a measurable phase region according to an optical signal of a light emitting unit according to an embodiment of the present invention.

Figure 10 (a) is a graph showing the normalized signal value corresponding to the actual phase, Figure 10 (b) is a graph corresponding to the actual phase by calculating the ^{arc tangent (tan -1) of Figure 10 (a).} It shows the measured phase value. As shown in (a) and (b) of Fig. 10, when the duty ratio of the optical signal is 25% or more (for example, the ideal case, the duty ratio is 50%, the duty ratio is 25%), the actual phase and measurement Since the phase has a one-to-one correspondence, it can be seen that the distance to the object can be measured in the entire phase region of the exposure period.

However, when the duty ratio is less than 25% (for example, when the duty ratio is 20% or the duty ratio is 10%), a section in which the actual phase and the measurement phase do not have a one-to-one correspondence may occur. That is, it may not be possible to measure a distance to an object in some of the phase regions of the entire phase region of the exposure period.

Therefore, in order to reduce the duty of the optical signal and improve the accuracy of distance measurement with respect to an object, the camera module 100 according to an embodiment of the present invention includes the light emitting unit 110 for each frame according to a plurality of sequences. The output phase of the output optical signal is shifted or the reception phase of the sensor unit 130 is shifted so that the distance to the object can be measured for the entire phase region of the exposure period.

11 is a diagram for explaining an output phase shift of a light emitting unit according to an exemplary embodiment of the present invention.

Referring to FIG. 11, the controller 140 may control the light emitting unit 110 to shift the output phase of the optical signal according to a plurality of sequences. For example, the control unit 140 may control the light emitting unit 110 to generate an optical signal of a reference phase (a part shown by a solid line in FIG. 11) according to a first sequence in a first exposure period. In addition, the control unit 140 controls the light-emitting unit 110 to generate an optical signal (a portion shown by a dotted line in FIG. 11) of an output phase in which a predetermined phase is shifted from the reference phase according to the second sequence in the second exposure period. Can be controlled. In this case, the second exposure period may mean an exposure period following the first exposure period. When the plurality of sequences includes the first sequence and the second sequence, the control unit 140 controls the light emitting unit 110 according to the second sequence, and then, in the third exposure period, the light emitting unit 110 is controlled according to the first sequence. ) Can be controlled. That is, a plurality of sequences may be sequentially repeated.

Table 1 is a table showing the phases of an optical signal when a plurality of sequences includes the first to the second-2 sequences.

Optical signal phase First sequence 0 2-1 sequence 0.16π 2-2 sequence 0.32π

According to Table 1, the control unit 140 controls the light emitting unit 110 to generate an optical signal of 0 phase (reference phase) according to the first sequence, and then the phase is changed by 0.16π according to the 2-1 sequence. The light emitting unit 110 may be controlled to generate a shifted optical signal. In addition, the controller 140 controls the light emitting unit 110 to generate an optical signal whose phase is shifted by 0.16π compared to the 2-1 sequence and whose phase is shifted by 0.32π compared to the first sequence according to the 2-2 sequence. I can.

12 is a diagram for explaining a phase shift of a reception of a sensor unit according to an embodiment of the present invention.

Referring to FIG. 12, the control unit 140 may control the sensor unit 130 to shift the reception phase of the optical signal, that is, the reception phase of the optical signal received by the sensor unit 130 according to a plurality of sequences. have. As described above, the sensor unit 130 may generate four image signals through four reference signals, that is, a demodulation signal, during one exposure period. The four reference signals may have a phase difference of 90 degrees. For example, the control unit 140 may control the sensor unit 130 to generate a reference signal of a reference phase according to a first sequence in a first exposure period. That is, the controller 140 may control the sensor unit 130 according to the reception phase corresponding to the reference phase according to the first sequence in the first exposure period. In addition, the control unit 140 may control the sensor unit 130 to generate a reference signal in which a predetermined phase is shifted from the reference phase according to the second sequence in the second exposure period. That is, the controller 140 may control the sensor unit 130 according to a reception phase corresponding to a phase in which a predetermined phase is shifted from the reference phase according to the second sequence in the second exposure period. In this case, the second exposure period may mean an exposure period following the first exposure period. When the plurality of sequences includes the first sequence and the second sequence, the controller 140 controls the sensor unit 130 according to the second sequence, and then, in the third exposure period, the sensor unit 130 ) Can be controlled. That is, a plurality of sequences may be sequentially repeated.

Table 2 is a table showing the phases of a demodulated signal when a plurality of sequences includes the first to third sequences.

First reception
Phase Second reception
Phase 3rd reception
Phase 4th reception
Phase First sequence 0 π/2 π 3π/2 2-1 sequence 0.16π π/2+0.16π π+0.16π 3π/2+0.16π 2-2 sequence 0.32π π/2+0.32π π+0.32π 3π/2+0.32π

According to Table 2, the control unit 140 controls the sensor unit 130 to receive an optical signal according to the reception phase of 0, π/2, π, and 3π/2 according to the first sequence, and then the 2-1 The sensor unit 130 may be controlled to receive an optical signal according to a reception phase of 0.16π, π/2+0.16π, π+0.16π, and 3π/2+0.16π according to the sequence. It can be seen that the phases of the first to fourth reception phases in the 2-1 sequence are shifted by 0.16π from the first to fourth reception phases in the first sequence. In addition, the controller 140 may control the sensor unit 130 according to the reception phase of 0.32π, π/2+0.32π, π+0.32π, and 3π/2+0.32π according to the 2-2 sequence. In the 2-2 sequence, the first to fourth reception phases are respectively shifted by 0.32π from the first to fourth reception phases of the first sequence, and the first to fourth reception phases of the 2-1 sequence, respectively. It can be seen that the phase shifted by 0.16π.

13 is a diagram for describing a measurable area for each phase according to an embodiment of the present invention.

When the duty ratio of the optical signal is 10%, when the distance is measured at the 0 degree phase (reference phase), the distance is measured only in the shaded part of FIG. 15A, and the distance may not be measured in other parts. . However, if the output phase of the light emitting unit 110 or the reception phase of the sensor unit 130 is shifted by a predetermined phase, the distance to the area not measured at the 0 degree phase as shown in FIGS. 15B to 15D Can be measured. Accordingly, when one depth image is generated through an image signal generated corresponding to each of a plurality of sequences, distance information corresponding to the entire phase may be obtained.

14 is a diagram illustrating an embodiment of setting a shift phase of a control unit according to an embodiment of the present invention.

14 is a diagram showing a case where the duty ratio is 10%, FIG. 14(a) is a graph showing a value of a normalized signal for an actual phase by a plurality of sequences, and FIG. 14(b) is a graph showing the actual phase. It is a graph showing the measurement phase for each of a plurality of sequences. Each shade shown in (b) of FIG. 14 refers to a phase region in which depth information can be obtained for each sequence.

As shown in FIG. 14, when the duty ratio is 10% and the phase shift is performed twice from the reference phase, distance information with respect to the object in the entire phase region of the exposure period can be obtained. Therefore, the optimal shift phase at a duty ratio of 10% is 0.16π.

In order to detect the optimum shift phase and measure the distance to the object, the controller 140 according to an embodiment of the present invention may set the shift phase based on the image signal generated in response to the first sequence. In more detail, the controller 140 may calculate a phase region in which a distance to an object can be measured through an image signal generated in response to the first sequence, and may set a shift phase according to the calculated phase region. The controller 140 may determine the number of second sequences according to the shift phase. In this case, the exposure period corresponding to the first sequence may be smaller than the exposure period corresponding to the second sequence. For example, the exposure period corresponding to the first sequence may be 50% of the exposure period corresponding to the second sequence.

On the other hand, in the case of the ToF camera module, the distance may not be measured for objects that are more than a certain distance. For example, in the case of measuring distance information for a specific object, the distance information may not be obtained because the background portion of the object may exceed the measurement distance. In the case of a portion where distance information cannot be obtained because the distance is too far as described above, distance information cannot be obtained even if a region that is not measured during the exposure period is measured through a phase shift due to a short duty ratio. As another example, since some of the objects are very close to the camera module 100, distance information may be obtained only with an image signal generated in response to the first sequence. In this case, even if an area that is not measured during the exposure period is measured through a phase shift, the obtained distance information does not change. That is, since various distance information is generated for each pixel according to a situation, there may be a case where it is not necessary to measure an unmeasured area during an exposure period through a phase shift.

For optimal driving in this situation, the control unit 140 according to the embodiment of the present invention controls the sensor unit 130 to shift the phase of the demodulation signal for the optical signal for each pixel of the sensor unit 130. I can. For example, in the second sequence, the sensor unit 130 is controlled to shift the phase of the demodulation signal for the first pixel by 0.16π, while the sensor unit 130 shifts the phase of the demodulation signal for the second pixel by 0.24π. The unit 130 can be controlled. In this case, the shift phase of the demodulation signal for each pixel may be set based on the image signal generated in response to the first sequence.

Although the embodiments have been described above, these are only examples and do not limit the present invention, and those of ordinary skill in the field to which the present invention belongs are not exemplified above without departing from the essential characteristics of the present embodiment. It will be seen that various modifications and applications are possible. For example, each component specifically shown in the embodiment can be modified and implemented. And differences related to these modifications and applications should be construed as being included in the scope of the present invention defined in the appended claims.

100: camera module
110: light emitting unit
120: light receiving unit
130: sensor unit
140: control unit
150: image processing unit
160: tilt part

Claims (9)

A light emitting unit that outputs an optical signal to the object;
A light-receiving unit that is output from the light-emitting unit and condenses the optical signal reflected on the object;
A sensor unit for receiving an optical signal condensed by the light receiving unit; And
A camera module including a control unit for controlling at least one of the light emitting unit and the sensor unit to shift at least one of an output phase of the light emitting unit and a reception phase of the sensor unit. The method of claim 1,
The duty ratio of the optical signal is,
Camera module set within a range greater than 0% and less than 25%. The method of claim 1,
The control unit,
Controlling the light emitting unit or the sensor unit according to a plurality of sequences,
The plurality of sequences,
A camera module comprising a first sequence corresponding to a reference phase and a second sequence shifted by a predetermined phase from the reference phase. The method of claim 3,
Each of the plurality of sequences,
Camera modules corresponding to different phases. The method of claim 3,
The exposure period of the optical signal corresponding to the first sequence,
A camera module that is smaller than an exposure period of an optical signal corresponding to the second sequence. The method of claim 3,
The control unit,
A camera module configured to set the phase of the second sequence based on an image signal generated in response to the first sequence. The method of claim 3,
The control unit,
A camera module in which each of the plurality of sequences is performed during an exposure period corresponding to one image frame. The method of claim 3,
The control unit,
A camera module controlling the sensor unit to shift the phase of the demodulated signal for the optical signal for each pixel of the sensor unit based on the image signal generated in response to the first sequence. The method of claim 1,
Camera module further comprising an image processing unit for generating a depth image by using a plurality of image signals corresponding to the plurality of sequences.

Images