JP7498703B2 - Method for acquiring depth information and camera module - Google Patents

Description

The present disclosure relates to a method and a camera module for obtaining depth information.

Devices that emit light and reflect it off an object to obtain information are used in a variety of fields. For example, from 3D cameras to distance measurement methods, technology that emits light to obtain information is used in a variety of ways.

For example, TOF (Time of Flight) is a term that refers to the principle of measuring distance by measuring the time difference between when light is emitted and when the light is reflected off an object and returned. TOF technology is easy to implement and is therefore used in a variety of fields, including aviation, shipbuilding, civil engineering, cameras, and surveying.

In connection with this, there is an increasing need for cameras with better hardware performance.

The present disclosure may provide a method and a camera module for acquiring depth information according to one or more embodiments. According to one embodiment, a method for acquiring a depth image of an object using light acquired during a first period and a second period in a camera module capable of acquiring depth information is disclosed. The camera module may acquire a depth image of an object using light acquired during a first period and a second period in a first receiving pixel and a second receiving pixel. The technical problem to be solved is not limited to the above technical problem, and may further include various technical problems within a scope obvious to a person skilled in the art.

The camera module according to the first aspect includes a light source that outputs light to an object; a receiver that receives light reflected from the object through a receiving pixel; and a processor that acquires depth information for the object using a phase difference between the light output by the light source and the light received by the receiver; the receiving pixels include a first receiving pixel and a second receiving pixel, the first receiving pixel receives light at a first phase point of a first period and a second phase point of a second period, respectively, and the second receiving pixel receives light at a third phase point of the first period and a fourth phase point of the second period, respectively, and the processor can acquire a depth image for the object using the information acquired during the first period and the second period.

Furthermore, the first to fourth phase points may be different from each other and correspond to any of 0°, 90°, 180°, and 270°.

Furthermore, the first receiving pixel and the second receiving pixel can be adjacent to each other.

Furthermore, the processor may interpolate information for the first receiving pixel at the third phase point with information acquired at the third phase point by pixels adjacent to the first receiving pixel.

The information for the third phase point may include charge amount information for the light received at the third phase point.

Additionally, the processor can apply super resolution techniques to increase resolution.

Furthermore, the receiver includes a first block and a second block obtained by dividing the receiving pixel, and the processor can obtain the depth information by using both the light received through the first block and the light received through the second block.

Furthermore, two pixels included in the first block and one of two pixels included in the second block may overlap.

The depth information acquisition method according to the second aspect may include the steps of: outputting light to an object; receiving light using a first receiving pixel at a first phase point of a first period and receiving light using a second receiving pixel at a third phase point of the first period; receiving light using a first receiving pixel at a second phase point of a second period and receiving light using a second receiving pixel at a fourth phase point of the second period; and acquiring a depth image for the object using information acquired during the first period and the second period.

Furthermore, the first to fourth phase points may be different from each other and correspond to any of 0°, 90°, 180°, and 270°.

Furthermore, the first receiving pixel and the second receiving pixel can be adjacent to each other.

The method may further include a step of interpolating depth information acquired by the first receiving pixel using light acquired during the first period with information on light acquired during the first period by one or more pixels diagonally adjacent to the first receiving pixel.

The third aspect can provide a computer-readable recording medium having a program recorded thereon for causing a computer to execute the method according to the second aspect.

The present disclosure may provide a method and a camera module for acquiring depth information according to one or more embodiments. According to one embodiment, a method for acquiring a depth image of an object using light acquired during a first period and a second period in a camera module capable of acquiring depth information is disclosed.

1 is a block diagram showing a configuration and operation of a camera module according to an embodiment; 1 illustrates a cross-sectional view of a camera module according to one embodiment. 1 illustrates an example in which a camera module according to an embodiment acquires a depth image for an object using light acquired by a first pixel and a second pixel during a first period and a second period. 4 is a timing chart for explaining the operation of the camera module shown in FIG. 3 over time. 11 is a diagram showing an example in which a phase signal of phase 0° is applied to a first pixel during a first period and a phase signal of phase 180° is applied to a second pixel, and a phase signal of phase 90° is applied to the first pixel during a second period and a phase signal of phase 270° is applied to the second pixel according to one embodiment. 6 is a timing chart for explaining the operation of the camera module in FIG. 5 over time. 11 is a diagram showing an example in which a phase signal of phase 0° is applied to a first pixel during a first period, a phase signal of phase 90° is applied to a second pixel, and a phase signal of phase 270° is applied to the first pixel during a second period, and a phase signal of phase 180° is applied to the second pixel according to one embodiment. 8 is a timing chart for explaining the operation of the camera module in FIG. 7 over time. 4 is a diagram illustrating an example in which a first pixel and a second pixel are horizontally adjacent to each other, according to an embodiment. 1 is a diagram illustrating an example in which a camera module operates with a first pixel and a third pixel horizontally adjacent to each other and a second pixel and a fourth pixel horizontally adjacent to each other, according to an embodiment. 1 is a diagram showing a method in which a camera module increases the resolution of an image using a super resolution technique. 1 is a diagram illustrating an example of how resolution is increased by a super-resolution method according to an embodiment; 1 is a diagram illustrating an example of increasing resolution by performing interpolation according to an embodiment; 1 is a flow chart illustrating a method for obtaining depth information for an object according to one embodiment.

The preferred embodiment of the present invention will now be described in detail with reference to the attached drawings.

However, the technical concept of the present invention is not limited to the embodiments described and may be embodied in various different forms, and one or more of the components of the embodiments may be selectively combined or substituted within the scope of the technical concept of the present invention.

Furthermore, terms (including technical and scientific terms) used in the embodiments of the present invention shall be interpreted as having a meaning that is commonly understood by a person of ordinary skill in the art to which the present invention belongs, unless expressly defined and described otherwise, and terms commonly used in conjunction with predefined terms should be interpreted in light of the contextual meaning of the relevant art.

Furthermore, the terms used in the embodiments of the present invention are intended to describe the embodiments and are not intended to limit the present invention.

In this specification, the singular can include the plural unless otherwise specified in the context, and when it says "A and/or at least one of B and C," it can include one or more of all possible combinations of A, B, and C.

Furthermore, in describing components of the embodiments of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. Such terms are used to distinguish the components from other components, and are not intended to limit the nature, order, or sequence of the components.

When a component is described as being 'coupled', 'bonded' or 'connected' to another component, this includes not only the case where the component is directly 'coupled', 'bonded' or 'connected' to the other component, but also the case where the component is 'coupled', 'bonded' or 'connected' to the other component via yet another component between the component and the other component.

Furthermore, when described as being formed or disposed "above" or "below" each component, "above" or "below" includes not only the case where two components are in direct contact with each other, but also the case where one or more additional components are formed or disposed between the two components. Furthermore, when described as "above" or "below", it can include not only the upper direction but also the lower direction based on one component.

Furthermore, the numerical values described below can be interpreted as being within a reasonable range of error. For example, a numerical value described as "1" can be interpreted as "1.01."

Below, an embodiment of the present invention will be described in detail with reference to the drawings. Hereinafter, 'light' is understood as a concept including 'optical signal', and 'signal' is understood as a concept including 'optical signal', and the terms may be used interchangeably.

Figure 1 is a block diagram showing the configuration and operation of a camera module 100 according to one embodiment.

As shown in FIG. 1, the camera module 100 may include a light source 1100, a processor 1000, and a receiver 120.

However, a person of ordinary skill in the art will understand that the camera module 100 may further include other general components in addition to the components illustrated in FIG. 1. For example, the camera module 100 may further include a diffuser through which light output from the light source passes, a light modulator (not shown) included in the light source 1100, or a memory (not shown) coupled to the processor 1000. The term "memory" may be broadly interpreted to include any electronic component capable of storing electronic information. The term memory may also refer to various types of processor-readable media such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage devices, registers, and the like.

If the processor 1000 can read information from/write information to the memory, the memory is said to be in electronic communication with the processor. Memory integrated into the processor 1000 is in electronic communication with the processor.

Furthermore, memory can be of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (such as SD or XD memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), etc. It can include at least one type of storage medium: memory, magnetic memory, magnetic disk, or optical disk.

Alternatively, a person having ordinary skill in the relevant art would understand that in other embodiments, some of the components illustrated in FIG. 1 may be omitted.

The light source 1100 according to one embodiment may output light. The light output from the light source 1100 may have a wavelength within a preset range.

The light source 1100 may be, for example, a light emitting diode (LED) or a laser diode (LD) capable of emitting light having an infrared wavelength or light having a near infrared (NIR) wavelength of about 850 nm that is invisible to the human eye for safety, but the wavelength band and type of light source are not limited. For example, the wavelength of the light output from the light source 1100 may be included in the visible light range or the ultraviolet light range.

Depending on the control signal received from the processor 1000, the light source 1100 can output light by performing, for example, amplitude modulation or phase modulation.

The light output from the light source 1100 to the object 130 in response to the control signal from the processor 1000 may have the form of a periodic continuous function having a preset period. For example, the light may have a specifically defined waveform such as a sine wave, a ramp wave, a square wave, a pulse wave, etc., but may also have a waveform of an undefined, general form.

The receiver 120 can receive light reflected by the object 130. The camera module 100 can obtain various information through the light received by the receiver 120.

The camera module 100 according to one embodiment can obtain information about the object 130 through the received light. For example, the processor 1000 can obtain various information about the object, such as the shape, size, color, and depth of the object 130.

The receiver 120 can distinguish between received light, which is acquired when light output from the light source 1100 is reflected by the object 130 among various light entering the receiver 120. For example, if the light source 1100 outputs light in the 750 nm to 950 nm range, the receiver 120 can selectively acquire light in the 750 nm to 950 nm range through filtering. Furthermore, the receiver 120 can acquire accurate information about the object 130 by selectively acquiring the received light corresponding to the light.

The camera module 100 according to one embodiment can extract depth information using a ToF function, and is therefore understood to be interchangeable with a ToF camera module or a ToF module in this disclosure.

The light source 1100 may generate light to be output and irradiate it on the object 130. At this time, the light source 1100 may generate and output light in the form of a pulse wave or a continuous wave. The continuous wave may be in the form of a sine wave or a square wave. By generating light in the form of a pulse wave or a continuous wave, the camera module 100 may determine a phase difference between the light output from the light source 1100 and the light reflected from the object and received by the camera module 100.

The light source 1100 can irradiate the generated light to the object 130 for a preset exposure period. The exposure period means one frame period. When generating multiple frames, the set exposure period can be repeated. For example, when the camera module 100 captures an object at 20 FPS, the exposure period is 1/20 [sec]. And, when generating 100 frames, the exposure period is repeated 100 times.

The light source 1100 can generate multiple lights having different frequencies. The light source 1100 can sequentially generate multiple lights having different frequencies. Alternatively, the light source 1100 can simultaneously generate multiple lights having different frequencies.

The light source 1100 according to one embodiment may output light to the object 1100 through output pixels. The light source 1100 may include output pixels, each of which may output light independently of one another. For example, the output pixels may output light of different intensities, may output light of different frequencies, may output light of different phases, and may output light having different delay times. Each output pixel may include a light emitting diode.

The receiver 120 according to one embodiment may receive light through a receiving pixel. The receiver 120 may receive reflected light obtained when light output from the light source 1100 is reflected by the object 130. The receiver 120 may include receiving pixels, each of which may receive light independently of the other. For example, the receiving pixels may receive light at different times and in different filtering methods.

The receiver 120 according to one embodiment may include a lens (not shown) and an image sensor. The lens may collect light reflected from the object 130 and transmit the light to the image sensor (not shown). The image sensor may receive the light and generate an electrical signal corresponding to the received light.

According to one embodiment, the light source 1100 can output light of different frequencies over time. For example, the light source 1100 can output light of frequency f1 during the first half of an exposure period and output light of frequency f2 during the second half of the exposure period.

According to one embodiment, some of the multiple light emitting diodes included in the light source 1100 may output light of frequency f1, and the remaining light emitting diodes may output light of frequency f2.

To control the multiple light emitting diodes included in the light source 1100, the light source 1100 may include an optical modulator.

The light source 1100 may generate light. The light generated by the light source 1100 may be infrared light having a wavelength of 770 to 3000 nm, or may be visible light having a wavelength of 380 to 770 nm. The light source 1100 may use a light emitting diode (LED), and may have a shape in which a plurality of light emitting diodes are arranged in a certain pattern. The light source 1100 may include an organic light emitting diode (OLED) or a laser diode (LD). Alternatively, the light source 1100 may be a VCSEL (Vertical Cavity Surface Emitting Laser). A VCSEL is a type of laser diode that converts electrical signals into light, and can use wavelengths of about 800 to 1000 nm, for example, about 850 nm or about 940 nm.

The light source 1100 can generate light in the form of a pulse wave or a continuous wave by repeatedly blinking (on/off) at regular time intervals. The regular time interval may be the frequency of the light. The blinking of the light source can be controlled by an optical modulator.

The optical modulator can control the on/off of the light source 1100 so that the light source 1100 generates light in the form of a continuous wave or a pulse wave. The optical modulator can control the light source 1100 to generate light in the form of a continuous wave or a pulse wave through frequency modulation, pulse modulation, etc.

The processor 1000 according to one embodiment may obtain depth information for the object 130 using a phase difference between the light output by the light source 1100 and the light received by the receiver 120. The receiver 120 may generate an electrical signal corresponding to each reference signal using a plurality of reference signals having different phase differences. The frequency of the reference signal may be determined to be the same as the frequency of the light output from the light source 1100. Thus, when the light source 1100 generates light at a plurality of frequencies, the receiver 120 may generate an electrical signal using a plurality of reference signals corresponding to each frequency. The electrical signal may include information regarding the charge amount or voltage corresponding to each reference signal.

In one embodiment, there may be four reference signals (C1 to C4). Each reference signal (C1 to C4) has the same frequency as the light output by the light source 1100, but may have a phase difference of 90 degrees from each other. One of the four reference signals (C1) may have the same phase as the light output by the light source 1100. The light reflected from the object 130 is delayed in phase by the distance the light output by the light source 1100 reflects back from the object 130.

The receiver 120 may generate signals Q1 to Q4 for each reference signal by mixing the received light with each reference signal. The receiver 120 may include an image sensor having a structure in which a plurality of pixels are arranged in a grid shape. The image sensor may be a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor. In addition, the image sensor may include a ToF sensor that receives infrared light reflected from an object and measures the distance using a time or phase difference.

Specifically, the processor 1000 can calculate the phase difference between the output light and the input light using the charge amount information of the electrical signal.

As described above, four electrical signals may be generated for each frequency of light output from the light source 1100. Therefore, the processor 1000 may determine the phase difference (td) between the light output from the light source 1100 and the light received by the receiver 120 using Equation 1 below.

Here, Q1 to Q4 may be the charge amount of each of the four electrical signals. Q1 may be the charge amount of the electrical signal corresponding to a reference signal having the same phase as the light output from the light source 1100, Q2 may be the charge amount of the electrical signal corresponding to a reference signal having a phase 180 degrees slower than the light output from the light source 1100, Q3 may be the charge amount of the electrical signal corresponding to a reference signal having a phase 90 degrees slower than the light output from the light source 1100, and Q4 may be the charge amount of the electrical signal corresponding to a reference signal having a phase 270 degrees slower than the light output from the light source 1100.

Then, the processor 1000 can determine the distance between the object 130 and the camera module 100 using a phase difference between the light output from the light source 1100 and the light received by the receiver 120. In this case, the processor 1000 according to an embodiment can determine the distance (d) between the object 130 and the camera module 100 using Equation 2.

where c is the speed of light and f is the frequency of the output light.

According to one embodiment, ToF IR images and depth images can be acquired from the camera module 100.

The processor 1000 according to one embodiment can obtain depth information for the object 130 by using the difference between the time when the light source 1100 outputs light and the time when the receiver 120 receives the light. The light source 1100 can obtain depth information by outputting light such as a laser or infrared light to the object 130, receiving the reflected light, and calculating the time difference.

Figure 2 illustrates a cross-sectional view of a camera module 100 according to one embodiment.

Referring to FIG. 2, the camera module 100 includes a lens assembly 200, an image sensor 250, and a printed circuit board 260. The processor 1000 of FIG. 1, etc., can be embodied in the printed circuit board 260. Although not shown, the light source 1100 of FIG. 1 can be disposed on the side of the image sensor 250 on the printed circuit board 260, or disposed outside the camera module 100, for example, on the side of the camera module 100, but is not limited thereto.

The lens assembly 200 may include a lens 230, a lens barrel 210, lens holders 221, 222, and an IR filter 240.

Lens 230 may be composed of multiple lenses or may be composed of a single lens. When lens 230 is composed of multiple lenses, each lens may be aligned based on a central axis to form an optical system. Here, the central axis may be the same as the optical axis of the optical system.

The lens barrel 210 may be coupled to the lens holders 221 and 222 to provide a space inside in which a lens can be accommodated. The lens barrel 210 may be rotationally coupled to one or more lenses, but this is merely an example and the lens may be coupled in other ways, such as by using an adhesive (e.g., an adhesive resin such as epoxy).

The lens holders 221 and 222 are coupled to the lens barrel 210 to support the lens barrel 210 and can be coupled to the printed circuit board 260 on which the image sensor 250 is mounted. The lens holders 221 and 222 can form a space in which the IR filter 240 can be attached to the lower part of the lens barrel 210. A spiral pattern can be formed on the inner peripheral surface of the lens holders 221 and 222, and the lens holders 221 and 222 can be rotatably coupled to the lens barrel 210, which has a similar spiral pattern formed on its outer peripheral surface. However, this is merely an example, and the lens holders 221 and 222 and the lens barrel 210 can be coupled to each other via an adhesive, or the lens holders 221 and 222 and the lens barrel 210 can be formed as an integral unit.

The lens holders 221 and 222 are divided into an upper holder 221 that is coupled to the lens barrel 210 and a lower holder 222 that is coupled to a printed circuit board 260 on which the image sensor 250 is mounted. The upper holder 221 and the lower holder 222 may be formed as an integral unit, or may be formed as separate structures and then fastened or coupled to each other, or may have separate and spaced structures. In this case, the diameter of the upper holder 221 is formed to be smaller than the diameter of the lower holder 222, but is not limited thereto.

The above example is merely an embodiment, and the lens 230 may be configured with other structures capable of collecting light incident on the camera module 100 and transmitting it to the image sensor 250.

The image sensor 250 can generate an electrical signal using the light focused through the lens 230.

The image sensor 250 can detect light input in synchronization with the blinking cycle of the light source 1100. Specifically, the image sensor 250 can detect light in phase and out phase with the light output from the light source 1100. That is, the image sensor 250 can repeatedly perform a step of absorbing light when the light source 1100 is on and a step of absorbing light when the light source 1100 is off.

The image sensor 250 can generate an electrical signal corresponding to each reference signal using a plurality of reference signals having different phase differences. The frequency of the reference signal can be determined to be the same as the frequency of the light output from the light source 1100. Therefore, when the light source 1100 generates light at a plurality of frequencies, the image sensor 250 can generate an electrical signal using a plurality of reference signals corresponding to each frequency. The electrical signal can include information regarding the amount of charge or voltage corresponding to each reference signal.

FIG. 3 illustrates an example in which the camera module 100 according to an embodiment acquires a depth image of an object using light acquired by the first pixel 310 and the second pixel 320 during the first and second periods. The first pixel 310 and the second pixel 320 may be receiving pixels.

Referring to FIG. 3, the camera module can sequentially acquire a first depth image (1), a second depth image (2), and a third depth image (3). Specifically, the camera module 100 may acquire a phase image for phase 0°, a phase image for phase 90°, a phase image for phase 180°, and a phase image for phase 270° in a first first period and a second period to acquire a first depth image (1), acquire a phase image for phase 0°, a phase image for phase 90°, a phase image for phase 180°, and a phase image for phase 270° in a second first period and a second period to acquire a second depth image (2), and acquire a phase image for phase 0°, a phase image for phase 90°, a phase image for phase 180°, and a phase image for phase 270° in a third first period and a second period to acquire a third depth image (3).

Specifically, the first pixel 310 included in the block 300 may acquire a phase image for phase 0° during the first period and acquire a phase image for phase 180° during the second period. The second pixel 320 included in the block 300 may acquire a phase image for phase 90° during the first period and acquire a phase image for phase 270° during the second period. However, the embodiment disclosed in FIG. 3 is not limited to this, and the phase image that the first pixel 310 or the second pixel 320 acquires during the first period or the second period may be determined according to a predetermined setting.

Because the strength of the signal received from when the pixel is opened once until it is closed is weak, the camera module 100 according to one embodiment may repeat the same process several times to acquire a depth image. For example, the block 300 may repeat the process of acquiring a phase image several times, for example, 100 times or more, to integrate or accumulate the signal to acquire a depth image.

Referring to FIG. 3, different phase signals may be applied to the pixels included in the block 300 during each period (T). For example, the block 300 may include a first pixel 310 and a second pixel 320, and a phase signal for phase 0° may be applied to the first pixel 310 during the first period, and a phase signal for phase 180° may be applied to the first pixel 310 during the second period. A phase signal for phase 90° may be applied to the second pixel 320 included in the block 300 during the first period, and a phase signal for phase 270° may be applied to the second pixel 320 during the second period, but is not limited thereto.

During one period (T), the intensity of the signal received by each pixel is weak, so the same process is repeated several times. The camera module 100 can repeat the period (T) in which a different phase signal is applied to each pixel several times, for example, 100 times or more, to integrate or accumulate the signal. The camera module 100 can read out information for phase 0° from the first pixel 310 in the first period and read out information for phase 90° from the second pixel 320. Furthermore, the first camera module 100 can read out information for phase 180° from the first pixel 310 in two periods and read out information for phase 270° from the second pixel 320. In addition, the first depth image (1) can be obtained using information for phase 0° and information for phase 180° obtained from the first pixel 310, and information for phase 90° and information for phase 270° obtained from the second pixel 320.

In this way, when a different phase signal is applied to each pixel included in one block 30 during each period (T) and the depth image is extracted using information on each phase obtained from each pixel, the time required to obtain the depth image can be reduced.

During each period (T), different phase signals are applied to adjacent pixels, and during each period (T), at least two of the sections in which a phase 0° phase signal or a phase 180° phase signal is applied to the first pixel and the sections in which a phase 90° phase signal or a phase 270° phase signal is applied to the second pixel overlap with each other. This reduces the time required to obtain one depth image compared to when the sections in which a phase 0° phase signal is applied, a phase 90° phase signal is applied, a phase 180° phase signal is applied, and a phase 270° phase signal are not applied.

Figure 4 is a timing diagram illustrating the operation of the camera module 100 in Figure 3 over time. The first pixel 310 and the second pixel 320 may be receiving pixels.

The first pixel 310 can receive light at a first phase point in the first period and a second phase point in the second period, and the second pixel 320 can receive light at a third phase point in the first period and a fourth phase point in the second period.

Referring to FIG. 4, in the first period, the first pixel 310 receives light with a delay of about 0°, so the first pixel 310 can receive a phase signal of phase 0°.

In the first period, the second pixel 320 receives light with a delay of about 90°, so the second pixel 320 can receive a phase signal with a phase of 90°.

In the second period, the first pixel 310 receives light with a delay of about 180°, so the first pixel 310 can receive a phase signal with a phase of 180°.

In the second period, the second pixel 320 receives light with a delay of about 270°, so the second pixel 320 can receive a phase signal of phase 270°.

However, without being limited to the embodiment disclosed in FIG. 4, the first to fourth phase points may be any combination different from each other corresponding to any of 0°, 90°, 180°, and 270°.

The first pixel 310 and the second pixel 320 may be adjacent to each other. As shown in FIG. 4, the first pixel 310 and the second pixel 320 may be adjacent vertically, but is not limited thereto. For example, the first pixel 310 and the second pixel 320 may be adjacent horizontally. Or, the first pixel 310 and the second pixel 320 may be adjacent diagonally.

Figure 5 is a diagram showing an example in which a phase 0° signal is applied to the first pixel 310 during a first period and a phase 180° signal is applied to the second pixel 320 during a second period, and a phase 90° signal is applied to the first pixel 310 and a phase 270° signal is applied to the second pixel 320 during a second period according to one embodiment. The first pixel 310, the second pixel 320, the third pixel 330, and the fourth pixel 340 may be receiving pixels.

Specifically, the first pixel 310 included in the block 300 may acquire a phase image for phase 0° during the first period and acquire a phase image for phase 90° during the second period. The second pixel 320 included in the block 300 may acquire a phase image for phase 180° during the first period and acquire a phase image for phase 270° during the second period. However, without being limited to the embodiment disclosed in FIG. 5, the first pixel 310 or the second pixel 320 may acquire a phase image in the first period or the second period according to a predetermined setting. Since the strength of the signal received from the pixel is weak until it is closed after being opened once, the camera module 100 according to one embodiment may repeat the same process several times to acquire a depth image. For example, block 300 can repeat the process of acquiring a phase image several times, for example, 100 times or more, to integrate or accumulate the signals to acquire a depth image.

The third pixel 330 may correspond to the first pixel 310, and the fourth pixel 340 may correspond to the second pixel 320. For example, the third pixel 330 may acquire a phase image for phase 0° during the first period together with the first pixel 310, and acquire a phase image for phase 90° during the second period. Furthermore, in this case, the fourth pixel 340 may acquire a phase image for phase 180° during the first period together with the second pixel 320, and acquire a phase image for phase 270° during the second period.

Figure 6 is a timing diagram that explains the operation of the camera module 100 shown in Figure 5 over time.

Referring to FIG. 6, in the first period, the first pixel 310 receives light with a delay of about 0°, so the first pixel 310 can receive a phase signal of phase 0°.

In the first period, the second pixel 320 receives light with a delay of about 180°, so the second pixel 320 can receive a phase signal with a phase of 180°.

In the first period, the third pixel 330 receives light with a delay of about 0°, so the third pixel 330 can receive a phase signal of phase 0°.

In the first period, the fourth pixel 340 receives light with a delay of about 180°, so the fourth pixel 340 can receive a phase signal with a phase of 180°.

In the second period, the first pixel 310 receives light with a delay of about 90°, so the first pixel 310 can receive a phase signal with a phase of 90°.

In the second period, the second pixel 320 receives light with a delay of about 270°, so the second pixel 320 can receive a phase signal of phase 270°.

In the second period, the third pixel 330 receives light with a delay of about 90°, so the third pixel 330 can receive a phase signal with a phase of 90°.

In the second period, the fourth pixel 340 receives light with a delay of about 270°, so the fourth pixel 340 can receive a phase signal of phase 270°.

Figure 7 shows an example in which a phase 0° signal is applied to the first pixel 310 during a first period and a phase 90° signal is applied to the second pixel 320, and a phase 270° signal is applied to the first pixel 310 during a second period and a phase 180° signal is applied to the second pixel 320, according to one embodiment.

Specifically, the first pixel 310 included in the block 300 may acquire a phase image for phase 0° during the first period and acquire a phase image for phase 270° during the second period. The second pixel 320 included in the block 300 may acquire a phase image for phase 90° during the first period and acquire a phase image for phase 180° during the second period. However, the embodiment disclosed in FIG. 7 is not limited to this, and the phase image that the first pixel 310 or the second pixel 320 acquires during the first period or the second period may be determined according to a predetermined setting.

Since the strength of the signal received from when the pixel is opened once until it is closed is weak, the camera module 100 according to one embodiment may repeat the same process several times to acquire a depth image. For example, the block 300 may repeat the process of acquiring a phase image several times, for example, 100 times or more, to integrate or accumulate the signal to acquire a depth image.

Figure 8 is a timing diagram that explains the operation of the camera module 100 shown in Figure 7 over time.

Referring to FIG. 8, in the first period, the first pixel 310 receives light with a delay of about 0°, so the first pixel 310 can receive a phase signal of phase 0°.

In the first period, the second pixel 320 receives light with a delay of about 90°, so the second pixel 320 can receive a phase signal with a phase of 90°.

In the second period, the first pixel 310 receives light with a delay of about 270°, so the first pixel 310 can receive a phase signal of phase 270°.

In the second period, the second pixel 320 receives light with a delay of about 180°, so the second pixel 320 can receive a phase signal with a phase of 180°.

Figure 9 shows an example in which a first pixel 310 and a second pixel 320 are horizontally adjacent in one embodiment.

As shown in FIG. 9, the relative positions of the first pixel 310 and the second pixel 320 can be determined in various ways. For specific details regarding the phase signals received by the first pixel 310 and the second pixel, refer to FIG. 3.

FIG. 10 is a diagram showing an example in which the camera module 100 operates with the first pixel 310 and the third pixel 330 horizontally adjacent, and the second pixel 320 and the fourth pixel 340 horizontally adjacent, according to one embodiment.

Referring to FIG. 10, unlike the case of FIG. 5, the first pixel 310 and the third pixel 330 are horizontally adjacent, and the second pixel 320 and the fourth pixel 340 are horizontally adjacent. That is, since pixels operating in the same manner are horizontally adjacent, the processor 1000 can control the pixels on a line-by-line basis. Since pixels in the same horizontal line operate in the same manner, the processor 1000 can control the pixels on a line-by-line basis, thereby reducing the complexity of the circuit. Furthermore, as in the case of FIG. 5, the block 300 includes two pixels 310, 320 with different operating methods.

Although FIG. 10 illustrates a case where pixels in a horizontal line perform the same operation, this is not limited to this. For example, pixels in the receiver 120 can be configured so that pixels in a vertical line perform the same operation.

Figure 11 is a diagram showing how the camera module 100 uses a super resolution technique to increase the resolution of an image.

Meanwhile, in one embodiment, the camera module 100 can use a Super Resolution (SR) technique to increase the resolution of the depth image. The SR technique can broadly refer to a method of obtaining a high-resolution image from multiple low-resolution images.

Specifically, the processor 1000 can acquire one piece of depth information per block. If one piece of depth information could be acquired per pixel, 25 pieces of depth information could be acquired from 25 pixels. However, if one piece of depth information could be acquired per block, the amount of information that can be acquired is reduced. Since one piece of depth information can be acquired by combining information on two pixels, the amount of information that can be acquired can be reduced by half in principle. For example, the processor 1000 can acquire two pieces of depth information from the first block 1110 and the second block 1120.

However, when combining information obtained from two pixels to obtain one piece of depth information, more information can be obtained if the pixels used are used redundantly. For example, the processor 1000 can use not only the first block 1110 and the second block 1120, but also the third block 1130. Furthermore, in some cases, one piece of depth information can be obtained through two adjacent pixels.

In FIG. 11, in one embodiment, a case is described in which the number of pixels included in a block is two and the number of overlapping pixels between overlapping blocks is one, but this is not limited thereto.

Figure 12 is a diagram illustrating an example of how resolution is increased using a super-resolution technique in one embodiment.

Referring to the first resolution drawing 1210, when information is obtained in pixel units, a resolution corresponding to the number of pixels can be obtained. However, when information is obtained in block units, when one pixel is used only once, the resolution can be reduced by the number of pixels contained in the block. For example, the resolution of the second resolution drawing 1220 is reduced by half compared to the first resolution drawing 1210. However, when the SR method as described above is used, the resolution can be increased by a significant amount, and even a higher resolution than that represented in the third resolution drawing 1230 can be achieved through an additional algorithm.

Figure 13 is a diagram illustrating an example of increasing resolution by performing interpolation according to one embodiment.

The phase of light received by the receiving pixel during the first period is shown in the first period drawing 1350, and the phase of light received by the receiving pixel during the second period is shown in the second period drawing 1360. The first pixel 1310 and the second pixel 1320 may be receiving pixels.

In the first period 1350, the processor 1000 according to one embodiment may obtain a phase 90° phase signal (information for a 90° phase point) at the first pixel 1310 by interpolating the phase 90° phase signals (information for a 90° phase point) obtained by the first pixel 1310 and the adjacent pixels 1311, 1312, 1313, and 1320. The first pixel 1310 may only obtain a phase 0° phase signal and a phase 180° phase signal in the first period and the second period, respectively, and may not obtain a phase 90° phase signal. However, the processor 1000 may obtain a phase 90° phase signal at the first pixel 1310 by interpolating the phase signals obtained by the adjacent pixels.

In the first period 1350, the processor 1000 according to one embodiment may obtain a phase 0° phase signal (information for the 0° phase time point) at the second pixel 1320 by interpolating the phase 0° phase signals (information for the 0° phase time point) obtained by the second pixel 1320 and the adjacent pixels 1321, 1322, 1123, and 1310. The second pixel 1320 may only obtain a phase 90° phase signal and a phase 270° phase signal in the first period and the second period, respectively, and may not obtain a phase 0° phase signal. However, the processor 1000 may obtain a phase 0° phase signal at the second pixel 1320 by interpolating the phase signals obtained by the adjacent pixels.

In the second period 1360, the processor 1000 according to one embodiment may obtain a phase 270° phase signal (information for the 270° phase time point) at the first pixel 1310 by interpolating the phase 270° phase signals (information for the 270° phase time point) obtained by the first pixel 1310 and the adjacent pixels 1311, 1312, 1313, and 1320. The first pixel 1310 may only obtain a phase 0° phase signal and a phase 180° phase signal in the first period and the second period, respectively, and may not obtain a phase 270° phase signal. However, the processor 1000 may obtain a phase 270° phase signal at the first pixel 1310 by interpolating the phase signals obtained by the adjacent pixels.

In the second period 1360, the processor 1000 according to one embodiment may obtain a phase 180° phase signal (information for a 180° phase point) at the second pixel 1320 by interpolating the phase 180° phase signals (information for a 180° phase point) obtained by the second pixel 1320 and the adjacent pixels 1321, 1322, 1123, and 1310. The second pixel 1320 may only obtain a phase 90° phase signal and a phase 270° phase signal in the first period and the second period, respectively, and may not obtain a phase 180° phase signal. However, the processor 1000 may obtain a phase 180° phase signal at the second pixel 1320 by interpolating the phase signals obtained by the adjacent pixels.

Figure 14 is a flow chart illustrating a method for obtaining depth information for an object according to one embodiment. Figure 14 can be understood with reference to the contents of Figures 1 to 13 described above.

In step S1310, the camera module 100 according to one embodiment outputs light to the object through output pixels. According to one embodiment, each output pixel may correspond to each receiving pixel.

In step S1320, the camera module 100 according to one embodiment receives light using a first receiving pixel at a first phase point of a first period, and receives light using a second receiving pixel at a third phase point of the first period. For example, in the first period, the first pixel may receive a phase signal of phase 0°, and the second pixel may receive a phase signal of phase 90°.

In step S1330, the camera module 100 according to one embodiment receives light using a first receiving pixel at a second phase point of the second period, and receives light using a second receiving pixel at a fourth phase point of the second period. For example, in the second period, the first pixel may receive a phase signal of phase 180°, and the second pixel may receive a phase signal of phase 270°. The first pixel and the second pixel may be receiving pixels.

However, without being limited to this embodiment, the first to fourth phase points may be any combination different from each other corresponding to any of 0°, 90°, 180°, and 270°.

In step S1340, the camera module 100 according to one embodiment acquires a depth image for the object using information acquired during the first and second periods. The processor 1000 can acquire one depth image using only the information acquired during the two periods.

In addition, although not shown in FIG. 13, a step of interpolating information for the third phase point at the first receiving pixel with information acquired at the third phase point by pixels adjacent to the first receiving pixel may be further included.

Meanwhile, the above-mentioned method can be created as a computer-executable program and can be implemented in a general-purpose digital computer that runs the program using a computer-readable recording medium. Furthermore, the data structure used in the above-mentioned method can be recorded in a computer-readable recording medium through various means. The computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, RAM, USB, floppy disk, hard disk, etc.) and optically readable media (e.g., CD-ROM, BD, etc.).

Although the embodiments of the present invention have been described above with reference to the attached drawings, those having ordinary skill in the art to which the present invention pertains should understand that the present invention can be embodied in other specific forms without changing its technical concept or essential features. Therefore, the embodiments described above should be understood to be illustrative in all respects and not limiting.

Claims (18)

A light source that emits light onto an object;
a receiver that receives light reflected from the object through a receiving pixel; and a processor that obtains depth information for the object using a phase difference between the light output from the light source and the light received by the receiver;
the receiving pixels include a first receiving pixel and a second receiving pixel;
The first receiving pixel receives light at a first phase time point of a first period and a second phase time point of a second period,
The second receiving pixel receives light at a third phase time point of the first period and a fourth phase time point of the second period,
the processor obtains a depth image for the object using information obtained during the first period and the second period;
The receiver includes a first block and a second block obtained by dividing the received pixel,
The processor obtains the depth information by using both the light received through the first block and the light received through the second block;
A camera module, wherein two pixels included in the first block and one of two pixels included in the second block overlap. The camera module of claim 1, wherein the first to fourth phase points are different from each other and correspond to any of 0°, 90°, 180°, and 270°. The camera module according to claim 1 or 2, wherein the first receiving pixel and the second receiving pixel are adjacent to each other. The camera module according to any one of claims 1 to 3, wherein the processor interpolates information at the first receiving pixel for the third phase time point with information acquired at the third phase time point by pixels adjacent to the first receiving pixel. A camera module according to any one of claims 1 to 4, wherein the information for the third phase time point includes charge amount information for the light received at the third phase time point. A camera module according to any one of claims 1 to 5, wherein the processor applies a super resolution technique to increase the resolution. The camera module according to any one of claims 1 to 6, wherein the light source outputs the light by performing amplitude modulation or phase modulation according to a control signal received from the processor. A camera module according to any one of claims 1 to 6, wherein the light output from the light source has the form of a periodic continuous function having a preset period according to a control signal from a processor. A camera module according to any one of claims 1 to 8, wherein the light source includes a plurality of output pixels, each of which outputs light independently of the others. The camera module according to claim 9, wherein the plurality of output pixels output light of different intensities, different frequencies, different phases, or different delay times. A light source that emits light onto an object;
an image sensor included in the receiver that receives light reflected from the object via receiving pixels;
a printed circuit board including a processor; and a lens barrel including a lens and coupled to the printed circuit board;
the processor obtains depth information for the object using a phase difference between the light output from the light source and the light received by the image sensor;
the receiving pixels include a first receiving pixel and a second receiving pixel;
The first receiving pixel receives light at a first phase time point of a first period and a second phase time point of a second period,
The second receiving pixel receives light at a third phase time point of the first period and a fourth phase time point of the second period,
the processor obtains a depth image for the object using information obtained during the first period and the second period;
The receiver includes a first block and a second block obtained by dividing the received pixel,
The processor obtains the depth information by using both the light received through the first block and the light received through the second block;
A camera module, wherein two pixels included in the first block and one of two pixels included in the second block overlap. The camera module according to claim 11, wherein the first to fourth phase points are different from each other and correspond to any one of 0°, 90°, 180°, and 270°. The camera module according to claim 11 or 12, wherein the first receiving pixel and the second receiving pixel are adjacent to each other. The camera module according to any one of claims 11 to 13, wherein the processor interpolates information at the first receiving pixel for the third phase time point with information acquired at the third phase time point by pixels adjacent to the first receiving pixel. outputting light onto an object;
receiving light using a first receiving pixel among the receiving pixels at a first phase time point of a first period, and receiving light using a second receiving pixel among the receiving pixels at a third phase time point of the first period;
receiving light using a first receiving pixel at a second phase time point of a second period and receiving light using a second receiving pixel at a fourth phase time point of the second period; and obtaining a depth image for the object using information obtained during the first period and the second period;
obtaining depth information by using both the light received through the first block and the light received through the second block in a first block and a second block obtained by dividing the receiving pixel;
Two pixels included in the first block and one of two pixels included in the second block overlap each other;
How to obtain depth information. The depth information acquisition method according to claim 15, wherein the first to fourth phase points are different from each other and correspond to any one of 0°, 90°, 180°, and 270°. The depth information acquisition method according to claim 15 or 16, wherein the first receiving pixel and the second receiving pixel are adjacent to each other. The depth information acquisition method according to any one of claims 15 to 17, comprising a step of interpolating information for the third phase time point at the first receiving pixel with information acquired at the third phase time point by pixels adjacent to the first receiving pixel.

Images