KR20220016574A - Camera system with offset pixel aperture for acquiring horizontal and vertical disparity - Google Patents

Abstract

Disclosed is a camera system with an OPA for acquiring horizontal disparity and vertical disparity. According to one embodiment, the camera system comprises: a single lens; and an image sensor consisting of at least one pixel array, wherein the at least one pixel array includes a plurality of pixels in a two-dimensional array. A light blocking film for forming an offset pixel aperture (OPA) is disposed in the upper part of each of the plurality of pixels. The OPAs are formed by each of the light blocking layers to maximize a distance apart from each other, and the OPAs include one pair of horizontal OPAs for acquiring disparity in a horizontal direction and one pair of vertical OPAs for acquiring disparity in a vertical direction.

Description

CAMERA SYSTEM WITH OFFSET PIXEL APERTURE FOR ACQUIRING HORIZONTAL AND VERTICAL DISPARITY

The following description relates to a camera system that calculates depth for a subject by using horizontal disparity and vertical disparity. Specifically, the camera system has a structure in which a baseline is increased to improve the range of the calculated depth. is a technique for

In a manner in which the depth of the subject indicated by the distance between the camera system (exactly, the image sensor included in the camera system) and the subject is calculated, blur between images acquired through a plurality of apertures of the camera system There is a method that uses change and a method that uses time difference.

In more detail, the conventional method using parallax is to configure a stereo camera system with two lenses and an image sensor, and then process an optical signal that is respectively introduced through the two lenses to obtain two images with parallax, Through the formula, the depth of the subject can be calculated based on the parallax between the two images.

However, since the conventional method using parallax constitutes one camera system with two lenses, it is difficult to miniaturize due to a large cross-sectional area, and the length of the baseline of the camera system is short, so that the parallax between images is small. There is a problem in that the range of depth is reduced.

Accordingly, there is a need for a technique for solving the disadvantages and problems of the conventional method of using parallax in a camera system.

One embodiment proposes a camera system having a structure to solve the problem of difficulty in miniaturization due to the large cross-sectional area and the problem that the range of the calculated depth is reduced due to the small parallax between the images.

Specifically, one embodiment proposes a structure for miniaturizing a camera system by minimizing the cross-sectional area of the camera system by applying Offset Pixel Apertures (OPAs) to each of the mono pixels included in the image sensor.

In addition, one embodiment forms a pair of horizontal OPAs for obtaining horizontal parallax and a pair of vertical OPAs for obtaining vertical parallax, and uses the horizontal OPAs and vertical OPAs to obtain horizontal parallax and vertical parallax By obtaining , we propose a structure that improves the range of the calculated depth.

According to one embodiment, a camera system comprises: a single lens; and at least one pixel array, wherein the at least one pixel array includes a plurality of pixels in a two-dimensional array, wherein an Offset Pixel Aperture (OPA) is disposed on each of the plurality of pixels. A light-shielding film for forming is disposed, and the OPAs are formed by each of the light-shielding films so that a distance apart from each other is maximized, and a pair of horizontal OPAs for obtaining a parallax in a horizontal direction and a pair of horizontal OPAs to obtain a parallax in a vertical direction A pair of vertical OPAs for

According to one aspect, each of the horizontal OPAs and each of the vertical OPAs may be formed on top of pixels located in a diagonal direction.

According to another aspect, each of the horizontal OPAs includes a left OPA biased in a left direction and a right OPA biased in a right direction with respect to the center of each pixel, and each of the vertical OPAs includes a It may be characterized in that it includes an upper OPA biased in an upward direction with respect to the center and a lower OPA biased in a downward direction.

According to another aspect, two OPAs located in a diagonal direction among the OPAs may have center positions shifted from each other with respect to each of the corresponding pixels.

According to another aspect, the image sensor may be a mono image sensor.

According to another aspect, a horizontal disparity between two images obtained through the two pixels in which the horizontal OPAs are formed and between two images obtained through the two pixels in which the vertical OPAs are formed It may be characterized by further comprising at least one processor for calculating the depth of the subject by using the vertical parallax.

According to another embodiment, the camera system includes a single lens; and an image sensor including a first pixel array and a second pixel array of 2×2 each including a first pixel, a second pixel, and two third pixels, wherein an OPA is disposed on each of the third pixels A light blocking film for forming an Offset Pixel Aperture is disposed, the first pixel array forms a pair of horizontal OPAs on the third pixels to obtain a parallax in a horizontal direction, and the second pixel array includes A pair of vertical OPAs for obtaining a parallax in a vertical direction are formed on the third pixels.

According to one aspect, each of the horizontal OPAs and each of the vertical OPAs may be formed on top of each of the third pixels located in a diagonal direction.

According to another aspect, each of the horizontal OPAs includes a left OPA biased in a left direction and a right OPA biased in a right direction with respect to the center of each of the third pixels, and each of the vertical OPAs includes the third It may be characterized in that it includes an upper OPA biased in an upward direction and a lower OPA biased in a downward direction with respect to the center of each pixel.

According to another aspect, a horizontal disparity between two images obtained through the third pixels in which the horizontal OPAs are formed and two images obtained through the third pixels in which the vertical OPAs are formed It may be characterized by further comprising at least one processor for calculating the depth of the subject by using the vertical parallax therebetween.

One embodiment may propose a camera system having a structure to solve the problem of difficulty in miniaturization due to the large cross-sectional area and the problem that the range of the calculated depth is reduced due to the small parallax between the images.

Specifically, embodiments may propose a structure for miniaturization by minimizing the cross-sectional area of a camera system by applying Offset Pixel Apertures (OPAs) to each of the mono pixels included in the image sensor.

In addition, one embodiment forms a pair of horizontal OPAs for obtaining horizontal parallax and a pair of vertical OPAs for obtaining vertical parallax, and uses the horizontal OPAs and vertical OPAs to obtain horizontal parallax and vertical parallax By obtaining , it is possible to propose a structure that improves the range of the calculated depth.

1 is a diagram for describing a relationship between a camera system according to an exemplary embodiment and a camera system in which an aperture is applied to a single lens.
2 is a diagram for explaining a depth calculation principle in a camera system according to an exemplary embodiment.
3 is a diagram for describing a constraint condition of a camera system according to an exemplary embodiment.
4 is a diagram illustrating a camera system according to an exemplary embodiment.
5 is a diagram for describing a pixel array included in an image sensor of a camera system according to an exemplary embodiment.
FIG. 6 is a diagram illustrating a lens aperture equivalent to a pixel array including pixels to which the OPA shown in FIG. 5 is applied.
7 is a diagram for describing pixel arrays included in an image sensor of a camera system according to another exemplary embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the examples. Also, like reference numerals in each figure denote like members.

In addition, the terms (Terminology) used in this specification are terms used to properly express a preferred embodiment of the present invention, which may vary depending on the intention of a viewer or operator or a custom in the field to which the present invention belongs. Accordingly, definitions of these terms should be made based on the content throughout this specification. For example, in this specification, the singular also includes the plural unless specifically stated in the phrase. Also, as used herein, “comprises” and/or “comprising” refers to a referenced component, step, operation and/or element being one or more other components, steps, operations and/or elements. The presence or addition of elements is not excluded.

Also, it should be understood that various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it should be understood that the position, arrangement, or configuration of individual components in each of the presented embodiments can be changed without departing from the spirit and scope of the present invention.

In order to obtain a 3D image to which the depth is applied, the depth of each pixel included in the 2D image should be calculated. At this time, in the conventional method of calculating the depth of each pixel included in the two-dimensional image, a laser is irradiated to a subject (object) to be photographed, and TOF (time of flight) is used to measure the return time of the light. method, a depth from stereo method that calculates depth using a parallax between images obtained from two or more camera systems, light passing through each of a plurality of apertures formed in a single lens in a single camera system A method of calculating depth using the parallax between images obtained by processing the signal (dual aperture; parallax method using DA), the optical signal passing through each of a plurality of apertures formed in a single lens in a single camera system There is a method of calculating the depth using the blur change between images obtained by processing, and the like.

A camera system according to an embodiment is based on a method using parallax between images among these conventional methods, but instead of forming a plurality of apertures in a single lens, pixels of a pixel array included in an image sensor By applying OPA (Offset Pixel Aperture) to the

In particular, the camera system according to an embodiment forms a pair of horizontal OPAs for acquiring horizontal parallax and a pair of vertical OPAs for acquiring vertical parallax, and horizontally using the horizontal OPAs and vertical OPAs. By acquiring the parallax and the vertical parallax, the range of the calculated depth is improved.

In the present invention, the OPA is a portion where the light-shielding film is not formed, and the OPA may be formed using the light-shielding film disposed on the pixel.

1 is a diagram for explaining a relationship between a camera system and a camera system to which an aperture is applied to a single lens according to an embodiment, and FIG. 2 is a diagram for explaining a depth calculation principle in the camera system according to an embodiment. , FIG. 3 is a diagram for explaining a constraint condition of a camera system according to an embodiment. Hereinafter, for convenience of explanation, FIG. 1 is described with reference to one pixel to which OPA is applied in the camera system according to an embodiment, but the camera system according to an embodiment includes two pixels to which OPA is applied as shown in FIG. It is based on an image sensor that

Referring to FIG. 1 , in any one pixel included in a pixel array of an image sensor in a camera system 110 according to an exemplary embodiment, a light blocking film 112 for forming an OPA 111 is disposed. At this time, O ₁ , which is the distance at which the center of the OPA 111 is offset from the center of the pixel, is the aperture in the camera system (conventional camera system) 120 in which the aperture 122 is formed in the single lens 121 . The center of (122) has a relationship proportional to O ₂ , which is a distance offset from the center of the single lens 121 .

Therefore, the OPA 111 having a central position shifted from the center of the pixel can be identical to the aperture 122 formed on the single lens 121 to have a central position shifted from the center of the single lens 121, For this reason, the camera system 110 according to an embodiment may calculate the depth of the subject by using the depth calculation formula in the conventional parallax method using DA.

및 초점 거리 f와 관련된 파라미터

(

는, 단일 렌즈(210)의 직경

가 OPA들(221, 231)이 서로 이격된 거리(정확하게는, 제1 OPA(221)의 중심과 제2 OPA(231)의 중심 사이의 거리)인

및 OPA들(221, 231)을 형성하기 위한 차광막들(222, 232)이 배치되는 픽셀들(220, 230) 각각의 높이인 h와 관련하여 비례하는 관계를 갖기 때문에(

), 픽셀들(220, 230) 각각의 높이인 h 및 OPA들(221, 231)이 서로 이격된 거리인

와 관련하여 비례하는 관계를 갖게 된다(

). 이에, 일 실시예에 따른 카메라 시스템(200)은 종래의 DA를 이용한 시차 방식에서의 깊이 산출 수식을 기반으로 아래의 식 1을 이용하여 피사체에 대한 깊이를 산출할 수 있다.In more detail, referring to FIG. 2 , the diameter of the single lens 210 in the camera system 200 according to an embodiment.

and parameters related to focal length f

(

is the diameter of a single lens 210

is the distance between the OPAs 221 and 231 (to be precise, the distance between the center of the first OPA 221 and the center of the second OPA 231 )

and since the light blocking films 222 and 232 for forming the OPAs 221 and 231 have a proportional relationship with respect to h, which is the height of each of the pixels 220 and 230 on which the OPAs 221 and 231 are disposed (

), h, which is the height of each of the pixels 220 and 230, and the distance at which the OPAs 221 and 231 are spaced apart from each other.

has a proportional relationship with

). Accordingly, the camera system 200 according to an embodiment may calculate the depth of the subject using Equation 1 below based on the depth calculation formula in the conventional parallax method using DA.

<Equation 1>

는 이미지 센서에 초점이 맞은 피사체 거리를 의미하며,

는 단일 렌즈(210)의 직경

및 초점 거리 f와 관련된 파라미터를 의미한다.In Equation 1, ds denotes a parallax between images acquired through two pixels 220 and 230 in which OPAs 221 and 231 are disposed in the image sensor of the camera system 200 according to an embodiment. , f means the focal length, a means the subject distance (the distance from the subject to the first main plane of the single lens of the camera system, which corresponds to the depth to the subject),

is the distance to the subject in focus on the image sensor,

is the diameter of a single lens 210

and a parameter related to the focal length f.

및 초점 거리 f와 관련된 파라미터인

가 카메라 시스템(200)에 포함되는 단일 렌즈의 f 넘버보다 작을 경우, 도 3에 도시된 310 경우와 같이 이미지 촬영 자체가 불가능하다. 따라서,

는 반드시 320 경우와 같이 단일 렌즈(210)의 f 넘버보다 커야 한다.At this time, the diameter of the single lens 210

and a parameter related to the focal length f,

When is smaller than the f number of a single lens included in the camera system 200 , as in case 310 illustrated in FIG. 3 , image capturing itself is impossible. thus,

must be greater than the f-number of the single lens 210 as in the case of 320.

), 아래의 식 3과 같이 표현될 수 있다.On the other hand, in Equation 2 below (d means pixel size in Equation 2) representing the disparity for each pixel derived from Equation 1, the focal length f is the size of the pixel array composed of pixels and the field of view (FOV) ) because it has a proportional relationship with (

), can be expressed as Equation 3 below.

<Equation 2>

<Equation 3>

는 카메라 시스템(200)에서 OPA들(221, 231)이 서로 이격된 거리 및 OPA들(221, 231)을 형성하기 위한 차광막들(222, 232)이 배치되는 픽셀들(220, 230) 각각의 높이와 관련하여 비례하는 관계를 갖는다(

).thus,

is a distance between the OPAs 221 and 231 in the camera system 200 and the pixels 220 and 230 in which light blocking films 222 and 232 for forming the OPAs 221 and 231 are disposed. It has a proportional relationship with respect to height (

).

가 반드시 320 경우와 같이 단일 렌즈(210)의 f 넘버보다 커야 하는 제약 조건은, OPA들(221, 231)이 서로 이격된 거리 및 OPA들(221, 231)이 적용된 픽셀들(220, 230) 각각의 높이와 관련된 오프셋 f 넘버(

)가 단일 렌즈(210)의 f 넘버보다 커야 하는 제약 조건으로 해석될 수 있다. 이하, 설명되는 일 실시예에 따른 카메라 시스템(200)은 해당 제약 조건을 만족시키는 가운데, OPA들(221, 231)이 서로 이격된 거리가 최대화되도록 하는 구조를 갖는다. 이에 대한 상세한 설명은 아래에서 기재하기로 한다. That is, the above

Constraints that must be greater than the f number of the single lens 210 as in the case of 320 are the distance at which the OPAs 221 and 231 are spaced apart from each other and the pixels 220 and 230 to which the OPAs 221 and 231 are applied. The offset f number associated with each height (

) can be interpreted as a constraint that must be greater than the f-number of the single lens 210 . Hereinafter, the camera system 200 according to an embodiment to be described has a structure that maximizes the distance between the OPAs 221 and 231 while satisfying the corresponding constraint. A detailed description thereof will be provided below.

4 is a diagram illustrating a camera system according to an exemplary embodiment.

배열의 4개의 픽셀들) 및 복수의 픽셀들의 상부에 공유되도록 배치되는 단일 마이크로 렌즈(421-1, 422-1)를 포함한다.Referring to FIG. 4 , a camera system 400 according to an exemplary embodiment includes a single lens 410 and an image sensor 420 positioned under the single lens 410 . Here, the image sensor 420 includes at least one pixel array 421 and 422 , and each of the at least one pixel array 421 and 422 includes a plurality of pixels (eg, a two-dimensional array).

4 pixels of the array) and a single micro lens 421-1, 422-1 disposed to be shared on top of the plurality of pixels.

The camera system 400 according to an embodiment of this structure applies OPAs to the pixels as described above, so that the parallax between images acquired through the pixels is similar to the case where DA is applied to a single lens 410 . It is possible to calculate the depth of the subject based on , and it is possible to achieve miniaturization by minimizing the cross-sectional area of the camera system compared to the case where DA is applied to the single lens 410 . In this case, the depth calculation operation may be performed by at least one processor (not shown in the drawing) further included in the camera system 400 .

In addition, the camera system 400 according to an embodiment allows a plurality of pixels to share a single micro lens 421-1 and 422-1, so that the baseline is lower than when a micro lens is provided in each of the plurality of pixels. In particular, by forming and placing the OPAs on the pixels maximizing the distance apart from each other, the baseline can be further increased. Accordingly, the camera system 400 may increase the baseline to increase the parallax between images and improve the range of the calculated depth.

A detailed structure of at least one pixel array 421 and 422 of the camera system 400 according to the exemplary embodiment will be described below.

5 is a diagram for explaining a pixel array included in an image sensor of a camera system according to an embodiment, and FIG. 6 is a lens aperture equivalent to the pixel array including the pixel to which the OPA shown in FIG. 5 is applied. It is a drawing. Here, FIG. 5 shows an exemplary view of a mono image sensor.

배열의 복수의 픽셀들(511, 512, 513, 514)을 포함하는 것으로 설명되나, 이에 제한되거나 한정되지 않고,

배열 또는

배열 등 다양한 2차원 배열의 복수의 픽셀들을 포함할 수 있다.Referring to FIG. 5 , a pixel array 500 included in an image sensor of a camera system according to an exemplary embodiment represents one pixel array 421 illustrated in FIG. 4 . Hereinafter, the pixel array 500 shares a single micro lens 500-1.

Although described as including, but not limited to, a plurality of pixels 511, 512, 513, 514 in an array,

array or

It may include a plurality of pixels in various two-dimensional arrays, such as an array.

Light blocking films 520 , 530 , 540 , 550 for forming OPAs 521 , 531 , 541 , 551 are disposed on each of the plurality of pixels 511 , 512 , 513 , and 514 of the pixel array 500 , , in particular, the OPAs 521 , 531 , 541 , and 551 are respectively formed using the light-blocking layers 520 , 530 , 540 , and 550 so that a distance from each other is maximized. For example, the OPA 521 of the first pixel 511 and the OPA 531 of the second pixel 512 use respective light blocking layers 520 and 530 in a diagonal direction to maximize the distance apart from each other. As shown in the figure, by being formed at the left end and the right end, it may mean horizontal OPAs for obtaining horizontal parallax, and the OPA 541 of the third pixel 513 and the OPA 551 of the fourth pixel 514 are , may mean vertical OPAs for obtaining vertical parallax by being formed at the upper and lower ends as shown in the drawing using the respective light blocking films 540 and 550 in the diagonal direction so that the distance from each other is maximized.

In addition, each of the pixels 511 , 512 , 513 , and 514 is a pixel array 500 such that the distance at which the OPAs 521 , 531 , 541 , 551 are spaced apart from each other is maximized in order to increase the baseline of the camera system. may be placed on the For example, using light blocking layers 520 and 530 on the pixels 511 and 512 diagonally positioned on the pixel array 500 so that the distance between the horizontal OPAs 521 and 531 is maximized. The light blocking layers 520 and 530 are used on the pixels 513 and 514 diagonally positioned on the pixel array 500 so that the vertical OPAs 541 and 551 are spaced apart from each other to maximize the distance. Accordingly, the horizontal OPAs 521 and 531 and the vertical OPAs 541 and 551 may be diagonally disposed in the pixel array 500 to maximize a distance from each other.

As described above, the camera system including the pixel array 500 to which the horizontal OPAs 521 and 531 and the vertical OPAs 541 and 551 are applied is identical to the equivalent lens aperture 610 shown in FIG. 6 . As a result, it is possible to have an increased baseline.

) 그리고 수직 OPA들(541, 551)이 서로 이격된 거리와 수직 OPA들(541, 551)이 적용되는 적어도 두 개의 픽셀들(513, 514) 각각의 높이와 관련된 오프셋 f 넘버(

)가 카메라 시스템의 단일 렌즈의 f 넘버보다 큰 제약 조건을 만족시키는 가운데, 수평 OPA들(521, 531)이 서로 이격된 거리와 수직 OPA들(541, 551)이 서로 이격된 거리가 최대화되도록 복수의 픽셀들(511, 512, 513, 514)에서 OPA들(521, 531, 541, 551)이 차광막들(520, 530, 540, 550)을 이용하여 각각 형성되는 위치가 결정될 수 있다. 여기서, 오프셋 f 넘버의 특성에 따르면, OPA들이 서로 이격된 거리 즉, 수평 OPA들(521, 531)이 서로 이격된 거리와 수직 OPA들(541, 551)이 서로 이격된 거리가 증가하더라도 OPA들(521, 531, 541, 551)이 적용되는 픽셀들(511, 512, 513, 514) 각각의 높이 역시 비례하여 증가된다면, 오프셋 f 넘버의 크기는 작아지지 않고 그대로 유지될 수 있다. 따라서, 일 실시예에 따른 카메라 시스템은 픽셀들(511, 512, 513, 514)을 수평 OPA들(521, 531)과 수직 OPA들(541, 551)이 서로 이격되는 거리가 최대화되도록 픽셀 어레이(500) 상에 배치하고, 서로 이격된 거리가 최대화되도록 차광막들(520, 530, 540, 550)을 이용하여 OPA들(521, 531, 541, 551)을 각각 형성하는 동시에, OPA들(521, 531, 541, 551)이 적용되는 픽셀들(511, 512, 513, 514) 각각의 높이 역시 증가시킴으로써, 오프셋 f 넘버가 단일 렌즈의 f 넘버보다 커야 하는 제약 조건을 만족시킬 수 있다.As such, in the plurality of pixels 511, 512, 513, and 514, the positions at which the OPAs 521, 531, 541, 551 are respectively formed using the light blocking films 520, 530, 540, 550 are determined. This may be performed to maximize the length of the baseline of the camera system below that satisfies the above-described constraint with reference to FIG. 3 . For example, the offset f number (

) and the offset f number (

) satisfies the constraint greater than the f number of a single lens of the camera system, and the horizontal OPAs 521 and 531 are spaced apart from each other and the vertical OPAs 541 and 551 are spaced apart from each other so that the distance is maximized. Positions at which the OPAs 521 , 531 , 541 , and 551 are respectively formed using the light blocking layers 520 , 530 , 540 and 550 in the pixels 511 , 512 , 513 , and 514 of may be determined. Here, according to the characteristic of the offset f number, the distance at which the OPAs are separated from each other, that is, the distance at which the horizontal OPAs 521 and 531 are separated from each other and the distance at which the vertical OPAs 541 and 551 are separated from each other increase, the OPAs If the height of each of the pixels 511 , 512 , 513 , and 514 to which ( 521 , 531 , 541 , 551 ) is applied is also proportionally increased, the size of the offset f-number may be maintained without being decreased. Accordingly, the camera system according to an exemplary embodiment includes pixels 511, 512, 513, and 514 in a pixel array ( 500) and at the same time forming OPAs 521, 531, 541, and 551 using light blocking films 520, 530, 540, and 550 so as to maximize a distance from each other, and at the same time, OPAs 521, By also increasing the height of each of the pixels 511 , 512 , 513 , and 514 to which the 531 , 541 , and 551 are applied, the constraint that the offset f-number must be greater than the f-number of a single lens may be satisfied.

In this case, the OPAs 521 , 531 , 541 , and 551 use light blocking layers 520 , 530 , 540 , and 550 to have a center position shifted from each other with respect to each of the pixels 511 , 512 , 513 , and 514 . each can be formed. For example, the first OPA 521 has a center shifted to the left with respect to the center of the first pixel 511 , and the second OPA 531 has a center shifted to the right with respect to the center of the second pixel 512 . By having a position, the first OPA 521 and the second OPA 531 may form horizontal OPAs having a center position shifted from each other, and the third OPA 541 is based on the center of the third pixel 513 . has a center position shifted upwardly, and the fourth OPA 551 has a central position shifted downward with respect to the center of the fourth pixel 514, so that the third OPA 541 and the fourth OPA 551 are It is possible to form vertical OPAs having center positions that are offset from each other. Accordingly, a parallax exists between the images obtained through the first pixel 511 and the second pixel 512 and between the images obtained through the third pixel 513 and the fourth pixel 514, Due to this, the camera system may calculate the depth of the subject based on the parallax between the images. That is, the camera system determines the horizontal parallax between the two images acquired through the first pixel 511 and the second pixel 512 in which the horizontal OPAs 521 and 531 are formed and the vertical OPAs 541 and 551 . The depth of the subject may be calculated using the vertical parallax between the two images acquired through the formed third pixel 513 and fourth pixel 514 .

A camera system including an image sensor and a single lens including at least one pixel array 500 having the above-described structure is a pixel 511 to which horizontal OPAs and vertical OPAs 521 , 531 , 541 , 551 are applied. , 512, 513, 514) can calculate the depth of the subject based on the horizontal and vertical parallax between images obtained through OPAs 521 by maximizing the distance that the plurality of pixels 511, 512, 513, 514 share a single micro lens and the OPAs 521, 531, 541, 551 are spaced apart from each other , 531 , 541 , 551 may be formed and disposed on the pixels 511 , 512 , 513 , and 514 to increase the baseline. In addition, the camera system constituting the mono image sensor forms horizontal OPAs and vertical OPAs in each of the pixel arrays, thereby acquiring horizontal parallax and vertical parallax in each of the pixel arrays, and through this, range can be improved.

7 is a diagram for explaining pixel arrays included in an image sensor of a camera system according to another exemplary embodiment, and is a diagram for explaining pixel arrays including pixels of a color image sensor, for example, RGB.

배열의 복수의 픽셀들(711, 712, 713, 714)을 포함하고, 제2 픽셀 어레이(720)는 단일 마이크로 렌즈를 공유하는

배열의 복수의 픽셀들(721, 722, 723, 724)을 포함하는 것으로 설명되나, 이에 제한되거나 한정되지 않고,

배열 또는

배열 등 다양한 2차원 배열의 복수의 픽셀들을 포함할 수 있다.Referring to FIG. 7 , a first pixel array 710 and a second pixel array 720 included in an image sensor of a camera system according to another exemplary embodiment are one pixel array 421 and 422 shown in FIG. 4 . indicates Hereinafter, the first pixel array 710 shares a single micro lens.

comprising a plurality of pixels 711, 712, 713, 714 of an array, wherein the second pixel array 720 shares a single micro lens.

Although described as including, but not limited to, a plurality of pixels 721, 722, 723, 724 of an array,

array or

It may include a plurality of pixels in various two-dimensional arrays, such as an array.

Light blocking films 730 and 740 for forming horizontal OPAs 731 and 741 on top of each of the two G pixels 712 and 713 among the RGGB pixels 711, 712, 713, and 714 of the first pixel array is disposed, and a light blocking film ( 750, 760) are arranged. In particular, each of the horizontal OPAs 731 and 741 and the vertical OPAs 751 and 761 is formed using the light blocking layers 730 , 740 , 750 , and 760 to maximize the distance from each other. For example, in the first pixel array 710 , the OPA 731 of the first G pixel 712 and the OPA 741 of the second G pixel 713 are each diagonally so that the distance apart from each other is maximized. It may be formed at the right end and the left end as shown in the drawing by using the light blocking films 730 and 740 of ), the OPA 761 may be formed at the upper end and the lower end as shown in the drawing by using each of the light blocking films 750 and 760 in the diagonal direction so that the distance from each other is maximized.

That is, in the first pixel array 710 , two images for acquiring a horizontal parallax are acquired using horizontal OPAs 731 and 741 formed on two G pixels, and the second pixel array 720 . In , by acquiring two images for obtaining vertical parallax using vertical OPAs 751 and 761 formed on two G pixels, the depth of the subject is determined using the horizontal parallax and vertical parallax obtained in this way. can be calculated. Accordingly, when the accuracy of calculating the depth using only one of the horizontal parallax and the vertical parallax in calculating the depth is reduced or the range of the depth is reduced, as in the present invention, the depth is calculated using the horizontal parallax and the vertical parallax. , it is possible to improve the accuracy of the calculated depth or the range of the depth. Of course, in the camera system according to another exemplary embodiment, the first pixel array 710 and the second pixel array 720 may be horizontally and vertically arranged in an array form.

As described above, the camera system including at least one image sensor including at least one pixel array 700 of the structure of FIG. 7 and a single lens includes pixels to which horizontal OPAs 731 and 741 are applied and vertical OPAs 751 , 761) can calculate the depth of the subject based on the horizontal and vertical parallax between the images obtained through the applied pixels, and minimize the cross-sectional area of the camera system compared to the case of applying DA to a single lens to promote miniaturization can do.

Although the color image sensor has been described with pixel arrays including pixels of RGBGB in FIG. 7, the color image sensor is not limited or limited to pixel arrays including pixels of RGB, and it is possible to include pixels of the same color in a pixel array. It can be applied to all curly image sensors including more than one.

Here, the other camera system is not limited or limited to the above-described content, and may include all the content described with reference to FIGS. 1 to 6 .

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the apparatus and components described in the embodiments may include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general purpose or special purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be embodied in any tangible machine, component, physical device, computer storage medium or device for interpretation by or providing instructions or data to the processing device. have. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. In this case, the medium may be to continuously store a program executable by a computer, or to temporarily store it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed over a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute other various software, and servers.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (9)

single lens; and
an image sensor comprising at least one pixel array, said at least one pixel array comprising a plurality of pixels in a two-dimensional array
including,
A light blocking film for forming an Offset Pixel Aperture (OPA) is disposed on each of the plurality of pixels,
The OPAs include a pair of horizontal OPAs for obtaining parallax in a horizontal direction and a pair of vertical OPAs for obtaining parallax in a vertical direction, which are formed by each of the light-shielding layers such that a distance apart from each other is maximized. which is a camera system. According to claim 1,
Each of the horizontal OPAs and each of the vertical OPAs,
The camera system, characterized in that formed on top of each of the pixels located in the diagonal direction. According to claim 1,
Each of the horizontal OPAs is
It includes a left OPA biased in the left direction and a right OPA biased in the right direction with respect to the center of each pixel,
Each of the vertical OPAs,
A camera system comprising: an upper OPA deflected in an upward direction and a lower OPA deflected in a downward direction with respect to the center of each pixel. According to claim 1,
The image sensor is
A camera system, characterized in that it is a mono image sensor. According to claim 1,
Using a horizontal disparity between two images obtained through the two pixels in which the horizontal OPAs are formed and a vertical disparity between two images obtained through the two pixels in which the vertical OPAs are formed, at least one processor to compute the depth for
A camera system further comprising a. single lens; and
An image sensor comprising a first pixel array and a second pixel array of 2×2 consisting of a first pixel, a second pixel and two third pixels
including,
A light blocking film for forming an Offset Pixel Aperture (OPA) is disposed on each of the third pixels,
the first pixel array
A pair of horizontal OPAs for obtaining a parallax in a horizontal direction are formed on the third pixels;
the second pixel array
and forming a pair of vertical OPAs on top of the third pixels to obtain parallax in a vertical direction. 7. The method of claim 6,
Each of the horizontal OPAs and each of the vertical OPAs,
The camera system, characterized in that formed on each of the third pixels located in the diagonal direction. 7. The method of claim 6,
Each of the horizontal OPAs is
a left OPA biased in a left direction and a right OPA biased in a right direction with respect to the center of each of the third pixels;
Each of the vertical OPAs,
and an upper OPA biased in an upward direction and a lower OPA biased in a downward direction with respect to the center of each of the third pixels. 7. The method of claim 6,
Using a horizontal disparity between two images obtained through the third pixels in which the horizontal OPAs are formed and a vertical disparity between two images obtained through the third pixels in which the vertical OPAs are formed at least one processor that calculates depth for the subject
A camera system further comprising a.

Images