KR102644944B1 - Image sensor and method to sense image - Google Patents

Abstract

An image sensing method and image sensor are provided. An image sensor according to an embodiment can restore a high-resolution image at high magnification by using sensing information corresponding to various viewing angles received through lens elements having the same focal length and various lens sizes.

Description

Image sensor and image sensing method {IMAGE SENSOR AND METHOD TO SENSE IMAGE}

Below, a technology for sensing images is provided.

Due to the development of optical technology and image processing technology, imaging devices are being used in a wide range of fields such as multimedia content, security, and recognition. For example, a photographing device may be mounted on a mobile device, camera, vehicle, or computer to capture data, recognize an object, or obtain data for controlling a device. The volume of the imaging device may be determined by the size of the lens, the focal length of the lens, and the size of the sensor. For example, the volume of the photographing device may be adjusted based on the size of the lens and sensor. As the size of the sensor decreases, the amount of light incident on the sensor may decrease. Accordingly, the resolution of the image may be lowered or shooting in a low-light environment may be difficult. To reduce the volume of the imaging device, a multi-lens composed of small lenses may be used. If the size of the lens is reduced, the focal length of the lens may be reduced. Therefore, the volume of the photographing device can be reduced through the use of multiple lenses.

An image sensor according to an embodiment includes a lens array including a plurality of lens elements having the same focal length; and a plurality of sensing elements spaced apart from the lens array by the focal length and sensing light passing through the lens array, wherein the lens size of at least one lens element among the plurality of lens elements is the lens of the other lens element. Size may vary.

The plurality of lens elements are classified into a plurality of lens groups according to lens size, lens elements belonging to the same lens group have the same lens size, and lens elements belonging to different lens groups have different lens sizes. You can.

The number of lens elements belonging to each of the plurality of lens groups may be determined based on the resolution shared by the plurality of lens elements.

Lens elements belonging to the same lens group may be arranged adjacent to each other.

One lens group among the plurality of lens groups may include a single lens element.

One lens element of the plurality of lens elements is disposed closer to the center position of the lens array than another lens element having a lens size larger than the lens size of the corresponding lens element, and one lens element of the plurality of lens elements is disposed closer to the center position of the lens array. The lens element may be disposed farther from the center position of the lens array than other lens elements having a smaller lens size than that of the lens element.

Each of the plurality of lens elements may be randomly arranged with respect to the plurality of sensing elements on a plane corresponding to the lens array.

Each of the plurality of lens elements may cover the same number of sensing elements.

At least one lens element among the plurality of lens elements may be arranged to be eccentric with respect to at least one sensing element among the plurality of sensing elements.

The image sensor may further include a processor that restores an image with higher resolution for a central area of the viewing angle corresponding to the lens array than for a surrounding area, based on sensing information sensed by the plurality of sensing elements.

The image sensor may further include a processor that acquires a compound eye vision image (CEV image) based on sensing information sensed through the plurality of sensing elements.

The processor may rearrange the compound eye field image based on light field information sensed by the plurality of sensing elements.

The processor may restore a scene image from the compound eye field image based on geometric structures between the plurality of sensing elements and the plurality of lens elements.

The processor may restore a scene image from the compound eye field image based on an image restoration model that has been trained before acquiring the compound eye field image.

The image sensor selects target information corresponding to a zoom factor specified by the user from the sensing information sensed through the plurality of sensing elements, and includes a processor that restores the scene image from the selected target information. More may be included.

The processor may select information corresponding to a viewing angle corresponding to the zoom magnification from the sensing information as the target information.

Each of the plurality of lens elements may refract the light to form a focal point at a point on the sensing array including the plurality of sensing elements.

Sensing elements belonging to the sensing area covered by each lens group can sense rays corresponding to the viewing angle of the corresponding lens group.

The image sensor may be implemented in a mobile terminal.

An image sensing method according to an embodiment includes: sensing light passing through a plurality of lens elements where the plurality of sensing elements have the same focal distance; And a step of a processor restoring a scene image based on the intensity of light sensed by the plurality of sensing elements, wherein the lens size of at least one lens element among the plurality of lens elements is the lens size of the other lens element. It may be different from

FIG. 1 is a diagram illustrating the general structure of an image sensor according to an embodiment.
FIG. 2 is a diagram illustrating a sensing element receiving light rays through a lens element according to an embodiment.
Figure 3 is a diagram showing the relationship between the number of sensing elements and the number of lens elements according to an embodiment.
FIG. 4 is a diagram illustrating the number and size of lens elements determined according to the focal length and desired viewing angle in a lens array according to an embodiment.
FIG. 5 is a diagram illustrating the arrangement of lens elements in a lens array according to an embodiment.
FIG. 6 is a diagram illustrating viewing angles of a scene sensed by sensing elements through lens elements arranged as described in FIG. 5.
7 to 10 are diagrams explaining the arrangement of lens elements according to another embodiment.
Figure 11 is a block diagram showing the structure of an image sensor according to an embodiment.
12 and 13 are diagrams illustrating examples of devices in which image sensors are implemented according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or limited by these examples. The same reference numerals in each drawing indicate the same members.

Various changes may be made to the embodiments described below. The embodiments described below are not intended to limit the embodiments, but should be understood to include all changes, equivalents, and substitutes therefor.

Terms used in the examples are merely used to describe specific examples and are not intended to limit the examples. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the embodiments belong. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless explicitly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No.

In addition, when describing with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted. In describing the embodiments, if it is determined that detailed descriptions of related known technologies may unnecessarily obscure the gist of the embodiments, the detailed descriptions are omitted.

FIG. 1 is a diagram illustrating the general structure of an image sensor according to an embodiment.

The quality of an image captured and restored by the image sensor 100 according to an embodiment may be determined by the number of sensing elements included in the sensing array 120 and the amount of light incident on the sensing elements. For example, the resolution of the image may be determined by the number of sensing elements included in the sensing array 120, and the sensitivity of the image may be determined by the amount of light incident on the sensing element. The amount of light incident on the sensing element may be determined based on the size of the sensing element. As the size of the sensing element increases, the amount of light incident on the sensing element may increase, and the dynamic range of the sensing array 120 may increase. Therefore, as the number of sensing elements included in the sensing array 120 increases, the image sensor 100 can capture high-resolution images, and as the size of the sensing elements increases, the image sensor 100 can capture high-sensitivity images in low light. It can work to your advantage.

The volume of the image sensor 100 may be determined by the focal length (f) of the lens element 111. More specifically, the volume of the image sensor 100 is determined by the gap between the lens element 111 and the sensing array 120. In order to collect the light 190 refracted by the lens element 111, the sensing array Since (120) must be located at the focal length (f) of the lens element 111, the lens element 111 and the sensing array 120 included in the image sensor 100 are located at the focal length (f) of the lens element 111. This is because they must be placed at a certain distance from each other. The focal length f of the lens element 111 is determined by the viewing angle of the image sensor 100 and the size of the lens element 111 (for example, the aperture radius of the lens element 111). For example, when the viewing angle is fixed, the focal length f becomes longer in proportion to the size of the lens element 111. Additionally, the size of the lens element 111 may be determined based on the size of the sensing array 120. For example, in order to capture an image with a certain viewing angle range, the size of the lens element 111 must be increased as the size of the sensing array 120 increases.

As described above, in order to increase image sensitivity while maintaining the viewing angle and image resolution, the volume of the image sensor 100 is increased. For example, to increase image sensitivity while maintaining image resolution, the size of each sensing element must be increased while maintaining the number of sensing elements included in the sensing array 120, so the size of the sensing array 120 must be increased. increases. At this time, in order to maintain the viewing angle, as the size of the sensing array 120 increases, the size of the lens element 111 increases and the focal length (f) of the lens element 111 becomes longer, so that the image sensor 100 The volume increases.

In order to reduce the volume of the image sensor 100, a design method of reducing the size of the sensing element while maintaining the resolution of the sensing array 120, or reducing the resolution of the sensing array 120 while maintaining the size of the sensing element. may be considered. However, when the size of the sensing element is reduced while maintaining the resolution of the sensing array 120, the size of the sensing array 120 is reduced and the focal length (f) of the lens element 111 is shortened, thereby reducing the image sensor 100. The volume can be reduced, but the sensitivity of the image decreases. In this case, low-light image quality may be reduced. In addition, when the resolution of the sensing array 120 is reduced while maintaining the size of the sensing element, the size of the sensing array 120 is reduced and the focal length (f) of the lens element 111 is shortened, so that the image sensor 100 The volume can be reduced, but the resolution of the image is reduced.

Referring to FIG. 1, the image sensor 100 includes a lens array 110 and a sensing array 120. The lens array 110 includes lens elements, and the sensing array 120 includes sensing elements.

According to one embodiment, as the size of each lens element included in the lens array 110 decreases, that is, as the number of lenses included in the same area on the lens array 110 increases, the lens element 111 The focal length f may be reduced, and the thickness of the image sensor 100 may be reduced. In this case, the image sensor 100 may restore a high-resolution image by rearranging and combining low-resolution images captured by each lens element 111. Accordingly, by dividing the lenses included in the lens array 110, a thin camera can be implemented.

Each individual lens element 111 of the lens array 110 may cover a certain sensing area 125 of the sensing array 120 corresponding to its lens size. The sensing area 125 covered by the lens element 111 in the sensing array 120 may be determined according to the lens size of the corresponding lens element 111. The sensing area 125 may represent an area on the sensing array 120 where rays of a certain viewing angle range arrive after passing through the corresponding lens element 111. The size of the sensing area 125 can be expressed as the distance D from the center of the sensing area 125 to the outermost point. In other words, light 190 passing through the individual lens element 111 may be incident on the sensing elements of the sensing array 120 included in the sensing area 125. Light 190 may include a plurality of rays. The ray 191 may correspond to the flow of photons 101.

According to one embodiment, the lens size may correspond to the diameter of the lens.

Each of the sensing elements of the sensing array 120 may generate sensing information based on the light ray 191 passing through the lenses of the lens array 110. For example, the sensing element 121 may generate sensing information based on the light ray 191 incident through the lens element 111. The image sensor 100 determines intensity information corresponding to an original signal regarding points included in the field of view of the image sensor 100, based on the sensing information output by the sensing array 120, The captured image can be restored based on the determined intensity information.

Additionally, the sensing element 121 may include a color filter for sensing an arbitrary color. The sensing element 121 may generate a color value corresponding to a specific color as sensing information. Each of the plurality of sensing elements constituting the sensing array 120 may be arranged to sense a color different from that of spatially adjacent adjacent sensing elements. However, the color filter and arrangement of the sensing element 121 are not limited to this.

When the diversity of the sensing information is sufficiently secured and a full rank relationship is formed between the original signal information and the sensing information corresponding to the points included in the field of view of the image sensor 100, the A captured image corresponding to the maximum resolution can be derived. Diversity of sensing information can be secured based on parameters of the image sensor 100, such as the number of lenses included in the lens array 110 and the number of sensing elements included in the sensing array 120.

For reference, although not shown in FIG. 1, the image sensor 100 may include a memory that stores an image restoration model used to restore an image, and a processor that restores the image using the image restoration model.

The image sensor 100 according to one embodiment can robustly restore an image even against various noise components that deviate from ideal conditions. Additionally, the image sensor 100 can restore an image regardless of any pattern (eg, Bayer pattern). Embodiments described below describe an operation in which the image sensor 100 restores a high-resolution image using multiple low-resolution images captured with multiple lenses. As will be described later, multiple low-resolution images may appear in the form of compound eye field images.

FIG. 2 is a diagram illustrating a sensing element receiving light rays through a lens element according to an embodiment.

As described above, the sensing array 220 may receive light rays corresponding to individual points 230 (X1 to X10). Light rays may be detected by the sensing array 220 through the lens array 210. For example, multiple light rays may be emitted from each of the individual points 230. Rays of light emitted from the same point can form a light field (LF). A light field emitted from an arbitrary point may be a field indicating the direction and intensity of light rays reflected from an arbitrary point on the subject. For example, rays emitted from the first point You can join the company. Rays emitted from each of the remaining points (X2 to X10) may also form a corresponding light field. Individual points 230 may be, for example, points on any object (eg, subject). The rays emitted from the individual points 230 may be rays reflected from an object, such as sunlight. For convenience of explanation, the lens array 210 in FIG. 2 is shown as including three lens elements and the sensing array as including ten sensing elements (S1 to S10), but the present invention is not limited thereto.

The sensing elements S1 to S10 may sense rays that have passed through a plurality of lens elements by overlapping them. For example, in the lens array 210 shown in FIG. 2, the focal distance from the lens element to the sensing array 220 may be reduced. Accordingly, the sensing element S1 may generate sensing information (eg, intensity value) in which rays emitted from the points X1 to X3 overlap. The image sensor needs to restore this overlapping sensing information in order to restore the image.

The sensing information generated by the sensing elements S1 to S10 shown in FIG. 2 is original signal information (e.g., intensity value).

[Equation 1]

In Equation 1 described above, S may represent a matrix indicating sensing information (eg, detected intensity value) sensed by an individual sensing element. X can indicate a matrix indicating a signal value corresponding to the rays (eg, color values of incident rays) corresponding to the rays incident with sensing elements (S1 to S10) from individual points. T is a transformation matrix and may represent the relationship between sensing information detected by the sensing elements S1 to S10 and signal information corresponding to incident light. In the structure shown in FIG. 2, rays, lenses, and sensing elements (S1 to S10) corresponding to individual points (X1 to X10) can be modeled as shown in Equation 2 below. In Equation 2 below, individual points (X1 to X10) may be modeled as being disposed at an infinite focal point from the image sensor. For example, the distance between individual points (X1 to X10) and the image sensor may be greater than the threshold distance.

[Equation 2]

In Equation 2 above, for convenience of explanation, signal information (eg, light intensity value) of the ray corresponding to each of the individual points (X1 to X10) is indicated as X1 to X10. Additionally, sensing information (eg, sensing intensity value) detected from the sensing elements S1 to S10 is also indicated as S1 to S10. According to one embodiment, sensing information (e.g., color information) corresponding to the sensing elements (S1 to S10) constituting the sensing array 220 and rays (X1 to X10) incident from individual points The relationship between the original signals (e.g., the transformation matrix described above) includes the arrangement between the lens elements and the sensing elements described above, the number of lens elements constituting the lens array 210, and the sensing elements constituting the sensing array 220. It may be determined based on the number of (S1 to S10), etc.

For reference, the above-mentioned equation 2 describes the case where the individual points (X1 to X10) are infinite focal points from the image sensor, and the individual points (X1 to X10) are finite focal points from the image sensor. When located, the original signal received from each sensing element may vary depending on the distance between the subject and the image sensor and the geometric structure of the image sensor.

Figure 3 is a diagram showing the relationship between the number of sensing elements and the number of lens elements according to an embodiment.

According to one embodiment, at least one lens element among the plurality of lens elements may be arranged to be eccentric with respect to at least one sensing element among the plurality of sensing elements. For example, a plurality of lens elements and a plurality of sensing elements may be arranged to be staggered with respect to each other. For example, the lens element may not cover an integer number of sensing elements, but may cover a non-integer number of sensing elements. The multi-lens array structure according to one embodiment may be implemented as a fractional alignment structure. The parameters of the multi-lens array structure are the number of sensing elements and the number of lens elements. The ratio P/L between the number L of lens elements and the number P of sensing elements may be determined as a real number. Each of the lens elements may cover the same number of sensing elements as the pixel offset corresponding to P/L. For reference, in FIG. 3, each lens element is shown to cover 10/3 sensing elements.

As described above, in the image sensor, the optical center axis (OCA) of each lens element may have a slightly different arrangement with respect to the sensing array. Accordingly, each lens element of the lens array receives different light field information. Since the direction of the chief ray of each lens element also changes, the image sensor can optically obtain more sensing information. Accordingly, the image sensor can restore a higher-resolution image through the various sensing information obtained in this way.

For example, when the lens elements included in the lens array have the same lens size, the number of lens elements included in the lens array and the number of sensing elements included in the sensing array that are relatively prime to each other are expressed in the equation below: 3 can be satisfied.

[Equation 3]

In Equation 3 above, R is an integer representing the resolution for one axis of the sensing array, P is an integer representing the number of sensing elements for one axis of the corresponding sensing array, and L is a lens element for one axis of the lens array. N, an integer representing the number of elements, may represent any natural number to satisfy Equation 3 described above. In Equation 3 above, R, P, and L may each represent the number of arbitrary axes on a two-dimensional plane. For example, resolution R may represent the resolution of the horizontal or vertical axis of information that can be sensed by an image sensor. The resolution R for the horizontal or vertical axis may correspond to the number P of sensing elements along the corresponding axis. L may represent the number of lens elements along the corresponding axis. Accordingly, the total resolution of the sensing array can be expressed as RxR, the total number of sensing elements included in the sensing array can be PxP, and the total number of lens elements included in the lens array can be LxL.

For example, Figure 3 may represent a cross-sectional view of the image sensor along the horizontal or vertical axis. For any one axis, the number of lens elements L is 3 and the number of sensing elements P is 10, so that they can satisfy a prime relationship. At this time, each lens element can cover 10/3 sensing elements for the corresponding axis.

In Figure 3, the first lens element can cover 0 to 3+1/3 sensing elements. The second lens element can cover 3+1/3 to 6+2/3 sensing elements. Likewise, the last lens element can cover 6+2/3 to 10 sensing elements. That is, each lens element can cover more of the portion corresponding to the disparity by 1/L (the number of lenses) in the sensing element.

According to one embodiment, according to the geometry of the lens array and the sensing array described above, the sensing elements covered by each lens element have light field information that is not the same as the light field information sensed by the sensing element covered by the other lens element. can be sensed. Light field information may represent information that is a combination of multiple light fields. For example, the first sensing element (S1) has a first light field at the first point (X1), a second light field at the second point (X2), and a third point (X3) in the structure shown in FIG. Light field information consisting of a combination of the third light fields can be sensed. On the other hand, the second sensing element S2 adjacent thereto can sense light field information consisting of a combination of the fourth light field, the fifth light field, and the sixth light field in the structure shown in FIG. 2. In this way, each sensing element can sense light field information that is different from light field information sensed by other sensing elements.

For example, the image sensor may rearrange the positions of pixels in a captured image based on correlation between light field information. For example, the image sensor may rearrange the pixels of a captured image (e.g., a compound eye field image) so that the pixels of the sensing elements that sense similar light field information in the plurality of sensing elements are adjacent to each other. .

For example, an image sensor may rearrange pixels indicating the intensity value of a signal sensed by an individual sensing element according to the similarity between the light field sensed by the corresponding sensing element and the light field information sensed by other sensing elements. . For example, the more the two light field information sensed by the two sensing elements include the same light fields, the higher the similarity between the two light field information may be.

According to one embodiment, the image sensor may determine light field information to be sensed by each sensing element, assuming that points reflecting light rays are disposed at infinite focus positions from the image sensor. For example, the image sensor, for each of a plurality of sensing elements, based on rays radiating from points farther than a threshold distance from the image sensor and the positional relationship between the plurality of sensing elements. Points that emit the light field sensed by the corresponding sensing element can be determined. Points farther than the critical distance may be referred to as points at the infinite focus position. The image sensor may rearrange pixels so that pixels corresponding to light fields emitted from spatially adjacent points are adjacent to each other.

For reference, in FIG. 2, individual points (X1 to X10) may be shown in the order in which they are adjacent to each other at infinite focal distance. For example, the first point (X1) may be adjacent to the second point (X2), and the second point (X2) may be adjacent to the first point (X1) and the third point (X3). Two points adjacent to each other may be, for example, spatially adjacent points in a subject.

Among the sensing elements 311 before rearrangement, the light field information sensed by the first sensing element (S1) and the light field information sensed by the eighth sensing element (S8) are both at the second viewpoint (X2) and the second viewpoint (X2). It may include a light field corresponding to 3 viewpoints (X3). Accordingly, the first sensing element S1 and the eighth sensing element S8 can sense similar light field information. If the above-described Equation 2 is rearranged so that pixels corresponding to similar light field information are adjacent to each other, it can be expressed as Equation 4 below.

[Equation 4]

Sensing elements 312 rearranged according to Equation 4 described above may be shown as shown in FIG. 3 . The first sensing element S1 may be covered by the first lens, the eighth sensing element S8 may be covered by the third lens, and the fifth sensing element S5 may be covered by the second lens. Since the sensing information sensed by each sensing element corresponds to the pixels constituting the image, the image sensor can rearrange the pixels so that the sensing information corresponding to the light rays that passed through different lenses are adjacent.

According to one embodiment, the image sensor applies rearrangement and restoration to images captured by the lens array and sensing array that satisfy each other's prerequisites and the above-mentioned equation 3, thereby creating a high-resolution scene with colors closer to the original scene. The image (scene image) can be restored.

FIG. 4 is a diagram illustrating the number and size of lens elements determined according to the focal length and desired viewing angle in a lens array according to an embodiment.

The image sensor according to one embodiment may include a lens array 410 and a sensing array 420 as described above. Lens array 410 may include a plurality of lens elements. The lens size of at least one lens element among the plurality of lens elements may be different from the lens size of the other lens elements.

For example, a plurality of lens elements may be classified into a plurality of lens groups according to lens size. A lens group may represent a group of lens elements classified by lens size. For example, lens elements belonging to the same lens group may have the same lens size, and lens elements belonging to different lens groups may have different lens sizes. An image sensor may include lens elements having various fields of view (FOV). The viewing angle of the lens element can be expressed as Equation 5 below.

[Equation 5]

In Equation 5 above, FOV may represent the viewing angle, D may represent the size of the sensing area covered by any lens element described above in FIG. 1, and F may represent the focal length 419.

The image sensor according to one embodiment may be implemented as a mobile terminal, and the focal length 419 F may be limited to meet the form factor of the mobile terminal. For example, the focal length (F) may be smaller than the thickness of the mobile terminal. To implement an image sensor in a mobile terminal, lens groups may be designed to have the same focal length 419 F. On the other hand, the viewing angle of each lens group may be designed to be different from the viewing angle of other lens groups.

For example, the image sensor may include a lens array 410 composed of lens groups having multiple viewing angles in order to obtain information about a distant subject. As described above, when the focal length 419 F is fixed for all lens elements, if the viewing angle FOV varies, the size (e.g., length D) of the sensing area covered by the lens element according to Equation 5 described above is It may vary. In other words, the lens array 410 may be designed so that as the zoom magnification of a lens group covering an arbitrary area on the sensing array 420 increases, the number of lens elements belonging to the lens group increases, and this lens array 410 ) Through the design of the lens group, the focal length of the lens group can be kept constant. Additionally, the lens size of the lens elements belonging to the corresponding lens group may decrease as the magnification of the lens group increases. The lens size of the lens elements belonging to the corresponding lens group may decrease even if the number of lens elements belonging to the corresponding group increases. Accordingly, the number of lens elements belonging to an arbitrary lens group may be determined based on the lens size according to the viewing angle and focal length of the corresponding lens group.

Additionally, in each lens group, the number of lens elements appearing along one axis of the lens array may satisfy Equation 6 below.

[Equation 6]

In Equation 6 above, R _i is the resolution of the image sensed through the ith lens group, L _i is the number of lens elements corresponding to the ith lens group appearing along one axis of the lens array, and N _i is an arbitrary natural number. , M may represent the number of lens groups included in the lens array 410. R _i , L _i , N _i , and M may be integers of 1 or more. i may be an integer between 1 and M. Therefore, the number of i-th lens elements belonging to the i-th lens group may be L _i x L _i .

Figure 4 shows a cross-sectional view of the lens array 410 with M = 3, and for convenience of explanation, the first lens group 411, the second lens group 412, and the third lens group ( 413). L ₁ may represent the number of lens elements corresponding to the first lens group 411 along one axis of the lens array, and in the lens array 410 shown in FIG. 4, L ₁ may be 3. L ₂ is the number of lens elements corresponding to the second lens group 412 along one axis of the lens array. In the lens array 410 shown in FIG. 4, L ₂ may be 9. L ₃ is the number of lens elements corresponding to the third lens group 413 along one axis of the lens array, and L ₃ may be 27. As described above in FIG. 3 , FIG. 4 also shows a cross-sectional view of the image sensor along one arbitrary axis, so L ₁ , L ₂ , and L ₃ may be the number of lens elements appearing along the axis. The number of first lens elements belonging to the first lens group 411 is 3x3 = 9, the number of second lens elements belonging to the second lens group 412 is 9x9 = 81, and the third lens element belonging to the third lens group is 9x9 = 81. The number of lens elements may be 27x27=729.

The first lens group 411 may transmit light rays within the first viewing angle 451 (eg, 77 degrees) to the first sensing element. The first sensing element may represent a sensing element covered by a first lens element belonging to the first lens group 411, and may receive light passing through the first lens element. The first sensing element may sense information corresponding to the first zoom magnification (eg, 1x). The second lens group 412 may transmit light rays within the second viewing angle 452 (eg, 30 degrees) to the second sensing element. The second sensing element may represent a sensing element covered by a second lens element belonging to the second lens group 412, and may receive light passing through the second lens element. The second sensing element may sense information corresponding to the second zoom magnification (eg, 3x). The third lens group 413 may transmit light rays within the third viewing angle 453 (eg, 10 degrees) to the third sensing element. The third sensing element may represent a sensing element covered by a third lens element belonging to the third lens group 413, and may receive light passing through the third lens element. The third sensing element may sense information corresponding to the third zoom magnification (eg, 9x).

According to one embodiment, the image sensor may be implemented as a single sensor or may be implemented as a plurality of sensors composed of the same sensing elements. At this time, each lens group in the image sensor may be designed to support the same resolution. For example, the sensing area covered by each lens group may include the same number of sensing elements as other groups. In this case, the number of lens elements along one axis of the lens array 410 may satisfy Equations 7 to 9 below.

[Equation 7]

[Equation 8]

[Equation 9]

In Equation 7 above, R may represent the resolution corresponding to one axis of the lens array shared by all lens groups. L _M may represent the number of lens elements corresponding to the Mth lens group along one axis of the lens array. L _i may represent the number of lens elements corresponding to the ith lens group along one axis of the lens array. N _M , N _i , and a _i can each represent any natural number. The number of lens elements belonging to each of the plurality of lens groups may be determined based on the resolution shared by the plurality of lens elements. Therefore, according to Equation 9 above, each lens group may have a resolution shared by all lens groups and include a number of lens elements that are coprime to the number of sensing elements covered by each lens element.

Therefore, the image sensor according to one embodiment applies rearrangement and restoration to the images captured by the lens array 410 and the sensing array 420 that satisfy Sowa Equation 9, thereby providing a closer approximation to the original scene. High-resolution images with color can be restored.

For example, an image sensor may include a color sensor (e.g., a sensor that senses images in the red, green, and blue channels in the visible light band) whose sensing elements are arranged in a Bayer pattern, and a color sensor whose sensing elements are arranged in a Bayer pattern. Images acquired from a camera using lens elements can be rearranged according to a similar light field order as described above. Since the image sensor can obtain a uniform color pattern, it can provide higher performance color interpolation.

FIG. 5 is a diagram illustrating the arrangement of lens elements in a lens array according to an embodiment.

According to one embodiment, lens elements belonging to the same lens group may be arranged adjacent to each other. In FIG. 5 , first lens elements of the first lens group 511 having the first lens size may be arranged adjacent to each other. Lens elements belonging to each of the remaining lens groups 512, 513, and 514 may also be arranged adjacent to each other.

FIG. 5 shows a top view of a structure in which M=4 and the lens array 510 includes four lens groups. The first lens group 511 has L ₁ = 2 and may include 2x2 = 4 lens elements. The second lens group 512 may include 4x4=16 lens elements, with L ₂ =4. Similarly, the third lens group 513 may include 64 lens elements, and the fourth lens group 514 may include 256 lens elements. For convenience of explanation in FIG. 5, the image sensor is shown as being implemented as a single sensor, but it is not limited to this, and the image sensor may be implemented as four image sensors including each lens group, and further, each other. It may also be implemented with different types of image sensors (for example, sensors with different sizes of individual sensing elements).

For reference, the processor of the image sensor may acquire a compound eye vision image (CEV image) based on sensing information sensed through a plurality of sensing elements. In this specification, the compound eye field image may represent an image obtained by overlapping the same or similar scene as observed through the compound eyes of an insect. For example, the image sensor may acquire a compound eye field image based on the intensity of light rays received from a plurality of sensing elements through a plurality of lenses arranged in an array.

For example, the processor may generate 2x2 first group images obtained by sensing light received through the first lens group 511 and 4x4 images obtained by sensing light received through the second lens group 512. A second group image, 8x8 third group images obtained by sensing light received through the third lens group 513, and 16x16 third group images obtained by sensing light received through the fourth lens group 514. A compound field of view image containing 4 group images can be created. As described above with reference to FIG. 3 , the processor may rearrange the compound eye field image based on light field information sensed by a plurality of sensing elements.

In FIG. 6 below, sensing information obtained about a scene through the lens array 510 shown in FIG. 5 is explained.

FIG. 6 is a diagram illustrating viewing angles of a scene sensed by sensing elements through lens elements arranged as described in FIG. 5.

The lens array according to one embodiment may receive light from outside the image sensor and transmit it to the sensing array. Each lens element of the lens array may have a different viewing angle depending on the lens size. For example, the first lens element belonging to the first lens group described in FIG. 5 has a first viewing angle, the second lens element belonging to the second lens group has a second viewing angle, and the third lens element belonging to the third lens group has a first viewing angle. 3 viewing angle, the fourth lens element belonging to the fourth lens group may have a fourth viewing angle.

Sensing elements belonging to the sensing area covered by each lens group can sense rays corresponding to the viewing angle of the corresponding lens group. Figure 6 shows sensing information sensed by the sensing element covered by the lens group for each viewing angle of each lens group.

The first sensing element covered by the first lens element may sense the first sensing information 611 by receiving rays corresponding to the first viewing angle through the first lens element. The first sensing information 611 may be sensed information about the area 610 corresponding to the first viewing angle. The second sensing element may sense the second sensing information 621 by receiving rays corresponding to the second viewing angle through the second lens element. The second sensing information 621 may be information sensed for the area 620 corresponding to the second viewing angle. The third sensing element may sense third sensing information 631 by receiving rays corresponding to the third viewing angle through the third lens element. The third sensing information 631 may be information sensed for the area 630 corresponding to the third viewing angle. Additionally, the fourth sensing element may sense fourth sensing information 641 by receiving rays corresponding to the fourth viewing angle through the fourth lens element. The fourth sensing information 641 may be information sensed for the area 640 corresponding to the fourth viewing angle.

As shown in FIG. 6, since sensing information is collected only by the first sensing element in the edge area of the area 610 corresponding to the entire viewing angle (e.g., the first viewing angle), the center area (e.g., the first viewing angle) The density of collected sensing information is low compared to the area 640 corresponding to the viewing angle of 4). In the central area, all sensing information sensed by the first to fourth sensing elements is collected, so the density of sensing information is high. Accordingly, the processor of the image sensor can restore an image with higher resolution for the central area of the viewing angle corresponding to the lens array than the surrounding area, based on sensing information sensed by the plurality of sensing elements.

Furthermore, in response to a zoom factor input from the user, the image sensor may generate an enlarged scene image based on the input zoom factor. For example, the image sensor may select target information corresponding to a zoom factor specified by the user from sensing information sensed through a plurality of sensing elements. The image sensor may select information corresponding to the viewing angle corresponding to the zoom magnification from the sensing information as target information. The image sensor can restore a scene image from selected target information. For example, when the zoom factor corresponding to the second viewing angle specified in the second lens group in FIG. 6 is input, the image sensor uses the sensing information sensed within the area 620 corresponding to the second viewing angle to view the scene. Images can be restored. The scene image restoration operation may include a pixel rearrangement operation according to the geometry of the image sensor or an operation of estimating an output image from the compound eye field image using an image restoration model.

Accordingly, when restoring an enlarged image (for example, an image whose viewing angle is narrowed as the zoom magnification increases), the viewing angle area provided by each lens element for the corresponding enlarged image may overlap. Due to this overlap of viewing angle areas, the image sensor can simultaneously use information from lenses of various structures (eg, various lens sizes) to restore a higher-resolution enlarged scene image.

For reference, Figure 6 assumes that rays reflected or emitted from points at infinite focus positions are received, and when rays are received from finite focus positions, the viewing angle range area of each lens group of the image sensor may vary.

7 to 10 are diagrams explaining the arrangement of lens elements according to another embodiment.

In the lens array 710 shown in FIG. 7, one lens group among a plurality of lens groups may include a single lens element. For example, the first lens group 711 may include a single lens element. Each of the remaining lens groups 712, 713, and 714 may include a plurality of lens elements as described above. Therefore, the image sensor restores the scene image so that both the image quality of the wide-angle image and the image quality of the image corresponding to the zoom ratio are improved based on the sensing information sensed through the lens array 710 of the structure shown in FIG. 7. can do.

In the lens array 810 shown in FIG. 8, lens elements can be arranged according to lens size. For example, one lens element among the plurality of lens elements may be disposed closer to the center position of the lens array 810 than another lens element having a lens size larger than that of the corresponding lens element. Additionally, one lens element among the plurality of lens elements may be disposed farther from the center position of the lens array 810 than another lens element having a lens size smaller than the lens size of the corresponding lens element.

For example, among the four lens groups, the first lens elements 811 belonging to the first lens group having the largest first lens size may be arranged at the edge of the lens array 810. The second lens elements 812 having a second lens size smaller than the first lens size may be arranged closer to the center position than the first lens elements 811. The third lens elements 813 having a third lens size smaller than the second lens size may be disposed closer to the center position than the second lens elements 812. The fourth lens elements 814 having a fourth lens size smaller than the third lens size may be disposed closest to the center position.

The lens array shown in FIG. 9 may include lens elements arranged identically or similarly to the lens array 810 shown in FIG. 8 . In FIG. 9, each of a plurality of lens elements according to an embodiment may cover the same number of sensing elements. For example, each of the first lens elements 911 may cover 4x4=16 sensing elements. The second lens elements 912, third lens elements 913, and fourth lens elements 914 may equally cover 16 sensing elements. In this case, the size of the sensing elements in the sensing array 920 may be designed so that the number of sensing elements covered by the lens elements is the same. For example, the second sensing size covered by the second lens elements 912 is greater than the first sensing size (e.g., pixel size) of the first sensing element covered by the first lens elements 911. The second sensing size of the element may be small. The third sensing size of the third sensing element may be smaller than the second sensing size, and the fourth sensing size of the fourth sensing element may be smaller than the third sensing size. In this specification, the sensing size of the sensing element may indicate, for example, the pixel pitch of the sensing element. Since the lens element that provides a small viewing angle also covers the same number of sensing elements as other lens elements, the degree of resolution degradation due to multiple lenses can be reduced and the quality of the restored scene image can be improved.

For reference, for convenience of explanation, the number of sensing elements covered by each lens element in FIG. 9 is shown as an integer, but it is not limited to this, and as described above in FIG. 3, each lens element is a fractional number. ) can cover the sensing elements.

In the lens array 1010 shown in FIG. 10, each of the plurality of lens elements 1011 may be randomly arranged with respect to the plurality of sensing elements on a plane corresponding to the lens array 1010. The plurality of lens elements 1011 may be configured in a number that satisfies Equation 9 described above for each lens size, and may be randomly arranged on the plane of the lens array 1010. With some of the randomly placed lens arrangements, the image sensor can restore a scene image with uniformly high resolution not only for wide angle but also for each of the zoom magnifications supported by the individual lens sizes.

Figure 11 is a block diagram showing the structure of an image sensor according to an embodiment.

According to one embodiment, the image sensor 1100 may include a lens array 1111, a sensing array 1120, and a processor 1130.

The lens array 1111 may include a plurality of lens elements. A plurality of lens elements may be disposed on a lens plane. The plurality of lens elements may all have the same or similar focal length. In the present specification, in a plurality of lens elements designed to have the same focal length, the focal distances of the plurality of lens elements may show a slight difference due to manufacturing tolerances. For example, the difference between the focal lengths of each of the plurality of lens elements may be less than a mutual threshold error. As described above, the lens size of at least one lens element among the plurality of lens elements may be different from the lens size of the other lens elements. Each of the plurality of lens elements may refract light to form a focal point at a point on the sensing array 1120 including a plurality of sensing elements.

Sensing array 1120 may include a plurality of sensing elements. A plurality of sensing elements may be disposed on a sensing plane parallel to the lens plane. A plurality of sensing elements may be arranged on a sensing plane spaced apart from the lens array 1111 by the focal distance. Each of the plurality of sensing elements can sense light passing through the lens array 1111. For example, each of the plurality of sensing elements may receive light passing through a lens element covering the corresponding sensing element.

The processor 1130 may restore a scene image based on the intensity of light sensed by a plurality of sensing elements. For example, the processor 1130 may acquire a compound eye field image based on sensing information sensed through a plurality of sensing elements. The processor 1130 may restore a scene image from a compound eye field image. The scene image is an image output by the image sensor 1100 and may represent a restored image identical to or similar to the original scene.

According to one embodiment, the processor 1130 may restore a scene image from a compound eye field image based on the geometric structure between a plurality of sensing elements and a plurality of lens elements. For example, as described above in FIG. 3, the processor 1130 selects a captured image (e.g., compound eye view) so that pixels of the sensing elements that sense similar light field information in the plurality of sensing elements are adjacent to each other. The pixels of an image can be rearranged.

According to another embodiment, the processor 1130 may restore a scene image from a compound eye field image based on an image restoration model that has been trained before acquiring the compound eye field image. The image restoration model is a model designed to output a scene image from an arbitrary compound eye field image and may be a machine learning structure. For example, an image restoration model may include a neural network structure. An image restoration model can be trained to produce a reference output image given as the ground truth from an arbitrary reference compound eye field image. However, the image restoration model is not limited to this.

12 and 13 are diagrams illustrating examples of devices in which image sensors are implemented according to an embodiment.

An image sensor according to an embodiment may be applied to various technical fields. The image sensor may be designed so that a lens array composed of a plurality of lenses and a sensor composed of a plurality of sensing elements are spaced apart at a relatively short focal length. Accordingly, the image sensor can be implemented as an ultra-thin camera while having a large sensor size for high-definition shooting. In this way, the image sensor can be implemented with a reduced thickness through a multi-lens array structure. The image sensor can be implemented with AP, FPGA, Chip, etc. and can be implemented as a camera's image signal processor.

Furthermore, by using a lens array that has an ultra-thin structure and has the same focal length and various lens sizes, the image sensor can obtain sensing information for a plurality of zoom magnifications. Accordingly, the image sensor can restore high-resolution scene images even for multiple zoom magnifications.

The image sensor may be implemented in a mobile terminal. Mobile terminals are terminals that are movable and not fixed to any geographical location, for example, portable devices (e.g., smart devices such as smartphones or tablets), artificial intelligence speakers, and vehicles. (vehicle), etc. may be included. 12 and 13 are examples of mobile terminals, and the mobile terminal is not limited thereto.

As shown in FIG. 12, for example, the image sensor 1210 may be applied to the front camera or rear camera of a smartphone. The image sensor 1210 has a structure that combines a large full frame sensor and a micro-lens array and can be applied to a mobile phone camera. For example, as shown in FIG. 12, an image sensor 1210 may be implemented as a front camera in the smart device 1200. The sensor of the image sensor 1210 may be implemented as a full frame, and the lens array may be implemented as a micro lens.

Additionally, it may be implemented for vehicles in a thin or curved structure. As shown in FIG. 13 , the image sensor 1310 may be implemented as a curved front or rear camera in the vehicle 1300. However, it is not limited to this, and the image sensor 1310 can be used for DSLR cameras, drones, CCTV, webcam cameras, 360-degree shooting cameras, cameras for movies and broadcasting, and VR/AR cameras. You can. Furthermore, the image sensor 1310 can be applied to various fields such as flexible/stretchable cameras, insect eye cameras, and contact lens type cameras.

Furthermore, the image sensor can also be applied to multi-frame super resolution image restoration, which increases resolution by using information from multiple frames captured in a video image of consecutive frames.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may perform an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. Computer-readable media may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with limited drawings as described above, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

1100: Image sensor
1111: Sensing array
1120: Lens array
1130: processor

Claims (20)

In mobile terminals,
a lens array including a plurality of lens elements having the same focal length; and
A plurality of sensing elements spaced apart from the lens array by the focal distance and sensing light passing through the lens array
Including,
The lens size of the first lens element among the plurality of lens elements is different from the lens size of the second lens element other than the first lens element among the plurality of lens elements,
Each of the plurality of lens elements has a viewing angle according to the following equation,
[Equation]

In the above equation, FOV represents the viewing angle, D represents the size of the sensing area covered by the lens element, and F represents the focal length of the lens element.
The viewing angle of the third lens element among the plurality of lens elements is
The viewing angle of the fourth lens element among the plurality of lens elements is different from that of the third lens element,
Each of the plurality of lens elements,
Covering a non-integer number of sensing elements among the plurality of sensing elements,
Mobile terminal. According to paragraph 1,
The plurality of lens elements are classified into a plurality of lens groups according to lens size,
Lens elements belonging to the same lens group have the same lens size,
Lens elements belonging to different lens groups have different lens sizes,
Mobile terminal. According to paragraph 2,
The number of lens elements belonging to each of the plurality of lens groups is determined based on the resolution shared by the plurality of lens elements,
Mobile terminal. According to paragraph 2,
Lens elements belonging to the same lens group are arranged adjacent to each other,
Mobile terminal. According to paragraph 2,
One lens group among the plurality of lens groups is,
Containing a single lens element,
Mobile terminal. According to paragraph 1,
One lens element of the plurality of lens elements is disposed closer to the center position of the lens array than another lens element having a lens size larger than the lens size of the lens element,
One lens element of the plurality of lens elements is disposed farther from the center position of the lens array than another lens element having a lens size smaller than the lens size of the lens element.
Mobile terminal. According to paragraph 1,
Each of the plurality of lens elements,
Randomly arranged with respect to the plurality of sensing elements on a plane corresponding to the lens array,
Mobile terminal. According to paragraph 1,
Each of the plurality of lens elements covers the same number of sensing elements,
Mobile terminal. According to paragraph 1,
At least one lens element among the plurality of lens elements,
arranged eccentrically with respect to at least one sensing element among the plurality of sensing elements,
Mobile terminal. According to paragraph 1,
A processor that restores an image with higher resolution for the central area of the viewing angle corresponding to the lens array than the surrounding area, based on the sensing information sensed by the plurality of sensing elements.
A mobile terminal further comprising: According to paragraph 1,
A processor that acquires a compound eye vision image (CEV image) based on sensing information sensed through the plurality of sensing elements.
A mobile terminal further comprising: According to clause 11,
The processor,
Rearranging the compound eye field image based on light field information sensed by the plurality of sensing elements,
Mobile terminal. According to clause 11,
The processor,
Reconstructing a scene image from the compound eye field image based on a geometry between the plurality of sensing elements and the plurality of lens elements,
Mobile terminal. According to clause 11,
The processor,
Restoring a scene image from the compound eye field image based on an image restoration model for which training was completed before acquiring the compound eye field image,
Mobile terminal. According to paragraph 1,
A processor that selects target information corresponding to a zoom factor specified by a user from sensing information sensed through the plurality of sensing elements and restores a scene image from the selected target information
A mobile terminal further comprising: According to clause 15,
The processor,
Selecting information corresponding to a viewing angle corresponding to the zoom magnification from the sensing information as the target information,
Mobile terminal. According to paragraph 1,
Each of the plurality of lens elements,
Refracting the light to form a focal point at a point on a sensing array including the plurality of sensing elements,
Mobile terminal. According to paragraph 1,
The sensing elements belonging to the sensing area covered by each lens group are:
Sensing rays corresponding to the viewing angle of the corresponding lens group,
Mobile terminal. delete In the image sensing method,
A plurality of sensing elements sensing light passing through a plurality of lens elements having the same focal length; and
A processor restoring a scene image based on the intensity of light sensed by the plurality of sensing elements.
Including,
The lens size of the first lens element among the plurality of lens elements is different from the lens size of the second lens element other than the first lens element among the plurality of lens elements,
Each of the plurality of lens elements has a viewing angle according to the following equation,
[Equation]

In the above equation, FOV represents the viewing angle, D represents the size of the sensing area covered by the lens element, and F represents the focal length of the lens element.
The viewing angle of the third lens element among the plurality of lens elements is
The viewing angle of the fourth lens element among the plurality of lens elements is different from that of the third lens element,
Each of the plurality of lens elements,
Covering a non-integer number of sensing elements among the plurality of sensing elements,
Image sensing method.

Images