KR100835058B1 - Image processing method for extending depth of field - Google Patents

Abstract

An image processing method is provided to extend depth of field for various camera modules without program amendment. An image processing method for extending depth of field includes the steps of: obtaining an edge profile curve for each of red, green, and blue channels included a Bayer format image(S31); calculating a blurring curve for each of the red, green, and blue channels by differentiating the edge profile curve(S32); deciding a blurring function for each of the red, green, and blue channels showing the blurring curve(S33); obtaining a correction coefficient for each of the red, green, and blue channels by using the blurring function(S341,S351,S361); and applying the correction coefficient to each of the red, green, and blue channels of the detected Bayer format image(S342,S352,S362).

Description

Image processing method for expanding depth of field {IMAGE PROCESSING METHOD FOR EXTENDING DEPTH OF FIELD}

1 is a block diagram of a conventional compact camera module.

2 is a diagram illustrating a spot size of each color channel of a Bayer format image measured by a conventional small camera module.

3 is a flowchart illustrating an image processing method for expanding a depth of field of the present invention.

4 is a block diagram illustrating a process of obtaining a blurring function using the blurring curve of the present invention.

5 is a diagram illustrating a Bayer format image obtained directly from an image sensor and a red channel, a green channel, and a blue channel extracted from a Bayer format image.

6 is a diagram illustrating a blurring curve obtained by differentiating a blurred edge profile and an edge profile.

7 is a flowchart illustrating a method of determining a correction coefficient such that the difference between the spot size and the target spot size of each corrected color channel is minimized.

FIG. 8 is a diagram illustrating a spot size of each color channel of a Bayer format image to which an image processing method for expanding depth of field according to the present invention is applied.

The present invention relates to an image processing method for expanding a depth of field of a camera module, and more particularly, to a processing technique for reducing spot size differences between respective color channels included in a Bayer format image detected from an image sensor. The present invention relates to an image processing method that can extend a depth of field of a camera module by extending a distance through which an image sharpness is secured.

Recently, a camera module applied to a mobile terminal such as a mobile phone is required to increase the number of pixels and to reduce the volume. In order to meet the demand for increasing the number of pixels, the pixel size of the image sensor is reduced, thereby realizing high pixel size in the image sensor having a limited area. In addition, in order to meet the demand for miniaturization, the number of lenses used in the camera module optical system is limited to three or less, and plastic is used instead of glass as the material of the lens for ease of mass production and cost reduction.

As the pixel size of the sensor decreases due to this trend, the number of pixels occupied by the spot of the beam focusing on the sensor surface increases, thereby increasing the chromatic aberration of the lens and causing a decrease in the resolution of the image. Constraints are inherent in expanding the depth of field.

1 is a block diagram showing a conventional small camera module. As shown in FIG. 1, the conventional miniature camera module blocks the light in the infrared (IR) region from the light passing through the optical system 11 and the light passing through the optical system 11. An infrared cut filter 12 that transmits light, an image sensor 13 that detects light in a visible light region passing through the infrared cut filter 12, and outputs a Bayer format image, and the image sensor 13 And an image signal processor (ISP) 14 for processing the Bayer format image generated by the < RTI ID = 0.0 > The image signal processor 14 performs color interpolation of each pixel using a Bayer image input from the image sensor 13 to determine the RGB value of each pixel, and to improve the image quality. A color processor 142 for correcting a color of the image, a gamma value, and a sharpness enhancer 143 for performing contrast stretching, sharpening, etc. on the image. Can be.

As shown in FIG. 1, the beams having the wavelengths of the red, green, and blue light that have passed through the optical system in the general compact camera module have different refractive indices according to the wavelengths, that is, the size of the spots formed in the image sensor 13, namely, The number of pixels occupied by the spots are all different. The number of pixels occupied by the spot formed by the beam for each color increases as the image sensor of a high pixel increases. 2 is a diagram illustrating such a difference in spot size.

2 shows the size of the spot measured by a 3 million pixel camera module using three pieces of APEL plastic material. In FIG. 2, the defocus value of the horizontal axis represents the degree of deviation of the image plane from the reference image plane, and the vertical axis represents the spot size of the red, green, and blue channels in each case. In FIG. 2, a position having a defocus value of 0 represents a reference image plane, and the reference image plane is defined as a plane in which a subject 1 m away focuses best. A negative defocus value indicates a case where the image plane is moved in a direction closer to the optical system, and a positive value indicates a case where the image plane is moved in a direction away from the optical system. Typically, the sensor is positioned on the reference image plane with a defocus value of zero in the camera module.

As shown in FIG. 2, the spot size of the green channel does not vary significantly within the depth of field (50 cm to infinity distance), while the spot size of the blue channel has the smallest spot size at close range (near 30 cm), but the spot goes farther away. As the size increases, the spot size of the red channel, on the contrary, is smaller in the distance, but the spot size increases rapidly as the distance increases. It can also be seen that the spot size of the Y channel image remains less than 4 pixels within the depth of field. This result indicates that the increase in the spot size of the blue channel at a long distance is the main cause of the decrease in sharpness, whereas the increase in the spot size of the red channel at the short distance is the main cause of the sharp decrease in sharpness.

Conventionally, the depth-of-field expansion method is known by designing a dedicated optical system with an extended depth of focus or by attaching a special optical phase mask to the lens so that the response of the camera module obtained from the sensor surface regardless of the distance of the subject is fixed. How to keep it The conventional depth-of-field expansion method applies a separate image processing technique for canceling the camera module response of a dedicated optical system or a phase mask with an extended depth of focus to an image captured with the camera module, thereby adjusting the depth of field of the camera module. Should be extended. However, conventional techniques for expanding depth of focus are much more sensitive to tilt or decenter and due to the expansion of the depth of focus, the focusing position in the optics design when assembling the camera module-the sensor from the last lens side of the optics. It is very difficult to find the position where the face is placed. In addition, the conventional technique using a phase mask has a problem in applying an image processing technique due to the inconvenience of having to manufacture a special mask separately and the characteristic that such a mask does not have cylindrical symmetry.

Accordingly, there is a need in the art for a technology that can extend the depth of field through an image processing algorithm without adding a separate physical component to a conventional small camera module.

The present invention has been proposed to solve the above-mentioned problems of the prior art, and an object thereof is a range of a subject distance for which image sharpness is secured through image processing for each color channel of a Bayer format image without a separate physical configuration. In other words, to provide an image processing method that can extend the depth of field.

The present invention as a technical configuration for achieving the above object,

a) placing a black and white line edge at a target distance to extend the depth of field, and imaging the black and white line edge using an image sensor to obtain a Bayer format image;

b) obtaining edge profile curves of the red, green and blue channels of the Bayer format image;

c) differentiating edge profile curves of the red, green, and blue channels to obtain blurring curves of the red, green, and blue channels;

d) determining the blurring function of the red channel, the blurring function of the green channel, and the blurring function of the blue channel, representing the blurring curves of the red channel, the green channel, and the blue channel; Function is a Gaussian distribution function;

e) a correction coefficient for the red channel, a correction coefficient for the green channel, and a correction coefficient for the red channel for correcting the blurring present in the red channel, the green channel, and the blue channel, respectively, by using the blurring functions of the red, green, and blue channels; Obtaining a correction coefficient for the blue channel; And

f) applying a correction coefficient for the red channel, a correction coefficient for the green channel, and a correction coefficient for the blue channel to the red, green, and blue channels of the Bayer format image detected by the image sensor, respectively.

It provides an image processing method for expanding the depth of field comprising a.

In a preferred embodiment of the present invention, the correction coefficient for the red channel, the correction coefficient for the green channel and the correction coefficient for the blue channel may have a parameter for adjusting the blurring correction amount for each color channel.

In this embodiment, g) the edge profile curves of the corrected red channel, green channel and blue channel are obtained by applying the correction coefficient for the red channel, the correction coefficient for the green channel and the correction coefficient for the blue channel, respectively. Obtaining spot sizes of the corrected red channel, spot sizes of the corrected green channel, and spot sizes of the corrected blue channel from the edge profile curves of the corrected red, green and blue channels; h) the spot size of the corrected red channel, the spot size of the corrected green channel, the spot size of the corrected blue channel and the target spot size of the preset red channel, the target spot size of the preset green channel, and the preset blue Comparing the target spot sizes of the channels, respectively; And i) adjusting the value of the parameter of the correction coefficient for the red channel until the spot size of the corrected red channel and the target spot size of the red channel are substantially equal to adjust the corrected correction coefficient for the red channel. Calculate the corrected correction coefficient for the green channel by adjusting a value of a parameter of the correction coefficient for the green channel until the spot size of the corrected green channel and the target spot size of the green channel are substantially the same. Calculate the adjusted correction coefficient for the blue channel by adjusting a value of a parameter of the correction coefficient for the blue channel until the spot size of the corrected blue channel and the target spot size of the blue channel are substantially the same. The method may further include calculating.

Preferably, each of the blurring functions representing the blurring curves for the red channel, the green channel, and the blue channel may be calculated by Equation 1 below.

[Equation 1]

(m _c -N _c ≤m≤m _x + N _x, n _{y c} -N ≤n≤n _c + N _y)

In Equation 1, Δx is the horizontal pixel spacing of the image sensor, Δy is the vertical pixel spacing of the image sensor, A is the height of the blurring function (the height of the horizontal blurring function and the height of the vertical blurring function M _c is the coordinate of the horizontal center pixel of the blurring function, n _c is the coordinate of the vertical center pixel of the blurring function, σ _x is the width of the horizontal blurring function, and σ _y is the vertical blurring. The width of the function.

In addition, the correction coefficient may be calculated by Equation 2 using a blurring function as in Equation 1.

[Equation 2]

In Equation 2, D denotes a correction coefficient, B denotes a Fourier transform of the function b of Equation 1, Δu is an inverse of Δx, Δv is an inverse of Δy, and k is a parameter.

According to an embodiment of the present invention, j) the spot size of the red channel, the green channel and the blue channel corrected in step f), the target spot size of the preset red channel, the target spot size of the preset green channel, and the preset Calculating and storing scores comparing the target spot sizes of the blue channels, respectively; k) modify the correction coefficient by changing the parameter, correct the red channel, green channel and blue channel of the Bayer format image detected by the image sensor by applying the corrected correction coefficient, and then correct the red Computing and storing scores comparing the spot sizes of the channels, the green channels and the blue channels, the target spot sizes of the red channels, the target spot sizes of the green channels, and the target spot sizes of the blue channels, respectively-this step Is performed repeatedly by changing the parameter; And l) determining a correction coefficient having a parameter for which a score having a minimum value among the stored scores is calculated as a final correction coefficient.

In this embodiment, the steps j) and k) are the horizontal spot sizes of the corrected red channel, green channel and blue channel, the target spot size of the red channel, the target spot size of the green channel and the blue channel. Calculating a horizontal score of the red channel, a horizontal score of the green channel, and a horizontal score of the blue channel, each comparing the target spot sizes; The vertical score of the red channel is compared with the corrected vertical spot sizes of the red, green and blue channels, the target spot size of the red channel, the target spot size of the green channel, and the target spot size of the blue channel, and green. Calculating a vertical score of the channel and a vertical score of the blue channel; And calculating the sum of the horizontal scores of the red, green, and blue channels and the vertical scores of the red, green, and blue channels, respectively, and storing the sums of the red, green, and blue channels. It may include a step.

As another technical configuration for achieving the above object of the present invention, the present invention,

a) positioning a black and white line edge at a target distance to extend the depth of field and at least one arbitrary distance between the target distance and the distance in the depth of field, and using an image sensor to place the black and white line edges at the plurality of distances; Photographing the image to obtain a plurality of Bayer format images;

b) obtaining edge profile curves of a red channel, a green channel, and a blue channel of each of the plurality of Bayer format images;

c) differentiate the edge profile curves of each of the red, green, and blue channels of each of the plurality of Bayer format images to obtain a plurality of blurring curves for the red channel, a plurality of blurring curves for the green channel, and a blue channel. Obtaining a plurality of blurring curves for;

d) determine a blurring function for a plurality of red channels representing the blurring curves for the plurality of red channels, and a blurring function for the plurality of green channels representing the blurring curves for the plurality of green channels Determining a blurring function for the plurality of blue channels representing the blurring curves for the plurality of blue channels, wherein the plurality of blurring functions for each color channel are Gaussian distribution functions;

e) apply a weight to each of the blurring functions for the plurality of red channels and add them all to obtain a total blurring function for the red channel, and apply a weight to each of the blurring functions for the plurality of green channels and add them all together Obtaining a total blurring function for the green channel, applying a weight to each of the blurring functions for the plurality of blue channels, and summing all of them to obtain a total blurring function for the blue channel;

f) a correction coefficient for the red channel, for the green channel, respectively for correcting the blurring present in the red channel, the green channel, and the blue channel by using the total blurring functions for the red channel, the green channel, and the blue channel, respectively; Obtaining a correction coefficient for the correction coefficient and the blue channel; And

g) applying a correction coefficient for the red channel, a correction coefficient for the green channel, and a correction coefficient for the blue channel to the red, green, and blue channels of the Bayer format image detected by the image sensor, respectively.

It provides an image processing method for expanding the depth of field comprising a.

Preferably, each of the blurring functions representing the blurring curves for the plurality of red, green, and blue channels may be determined as in Equation 3 below, and the total blur of the red, green, and blue channels may be determined. The ring curve may be determined as in Equation 4 below.

[Equation 3]

(m _c -N _c ≤m≤m _x + N _x, n _{y c} -N ≤n≤n _c + N _y)

[Equation 4]

In Equation 3, b _p is a blurring function at the distance p, Δx is the horizontal pixel spacing of the image sensor, Δy is the vertical pixel spacing of the image sensor, A _p is the height of the blurring function at the distance p (horizontal The average of the height of the directional blurring function and the height of the vertical blurring function), m _c is the coordinate of the horizontal center pixel of the blurring function at distance p, and n _c is the The coordinate, σ _xp, is the width of the horizontal blurring function at distance p, and σ _yp is the width of the vertical blurring function at distance p. When the black and white line edges are photographed at each distance p, the positions of the edges are the same. Therefore, the horizontal and vertical center coordinates of the blurring function are the same at each distance p.

In addition, in Equation 4, b represents an overall blurring function, and w _p represents a weight.

Hereinafter, with reference to the accompanying drawings will be described in detail an embodiment of the present invention. However, embodiments of the present invention may be modified in various other forms, and the scope of the present invention is not limited to the embodiments described below. Embodiment of this invention is provided in order to demonstrate this invention more completely to the person skilled in the art to which this invention belongs. In addition, in the description of the present invention, terms defined are defined in consideration of functions in the present invention, which may vary according to the intention or convention of those skilled in the art, and thus limit the technical components of the present invention. It should not be understood as meaning.

3 is a flowchart illustrating an image processing method for expanding a depth of field of the present invention.

Referring to FIG. 3, the image processing method for expanding the depth of field according to the present invention includes obtaining edge profile curves of red, green, and blue channels included in a Bayer format image outputting a black and white line edge by an image sensor. Obtaining a blurring curve of the red channel, the green channel, and the blue channel by differentiating the edge profile curves of the red channel, the green channel, and the blue channel (S32); Determining a blurring function of the red channel, the blurring function of the green channel, and the blurring function of the blue channel, which represent the blurring curve of the blue channel (S33), and the blurring of the red channel, the green channel, and the blue channel Correction coefficient for the red channel, correction channel for the green channel, respectively, for correcting blurring present in the red, green, and blue channels by using a function And obtaining correction coefficients for the blue channel (S341, S351, S361) and correction coefficients for the red channel, and correction for the green channel in the red, green, and blue channels of the Bayer format image detected by the image sensor. And applying the coefficient and the correction coefficient for the blue channel (S342, S352, S362), respectively.

In addition, the present invention, in addition to the above-described configuration, the edge of the red channel, green channel and blue channel corrected by applying the correction coefficient for the red channel, the correction coefficient for the green channel and the correction coefficient for the blue channel, respectively Obtaining a profile curve and obtaining a spot size of the corrected red channel, a spot size of the corrected green channel, and a spot size of the corrected blue channel from the edge profile curves of the corrected red, green, and blue channels (S342, S352 and S362), the spot size of the corrected red channel, the spot size of the corrected green channel, the spot size of the corrected blue channel and the target spot size of the preset red channel, and the target spot size of the preset green channel And comparing target spot sizes of preset blue channels (S343, S353, and S363), and spot sizes of the corrected red channel and the red color. Adjust the value of the parameter of the correction coefficient for the red channel until the target spot size of the null is substantially the same (S344) to calculate the corrected correction coefficient for the red channel (S345), and the corrected green channel The modified correction coefficient for the green channel is calculated by adjusting the value of the parameter of the correction coefficient for the green channel until the spot size of and the target spot size of the green channel are substantially the same (S354). The adjusted correction coefficient for the blue channel is adjusted by adjusting a value of a parameter of the correction coefficient for the blue channel until the spot size of the corrected blue channel and the target spot size of the blue channel are substantially the same (S364). It may further include the step (S365).

First, in one embodiment of the present invention, the step (S31) of obtaining the edge profile curves of the red channel, the green channel, and the blue channel included in the Bayer format image which captures and outputs the black and white line edges in the image sensor, the depth of field Place a black and white line edge at a target distance to extend, capture an image of the black and white line edge using an image sensor, acquire one Bayer format image, and then red and green channels included in the obtained Bayer format image. And calculating an edge profile curve of the blue channel. This embodiment can extend the depth of field by correcting the spot sizes of the red channel, the green channel, and the blue channel in the image acquired at the target distance to extend the depth of field by the target spot size at the predetermined target distance. have.

In another embodiment, the step (S31) of calculating the edge profile curves of the red, green, and blue channels included in the Bayer format image that captures and outputs the black and white line edges in the image sensor may extend the depth of field. Place a black and white line edge at a desired target distance and at least one arbitrary distance between the target distance and a distance in the depth of field, and image the black and white line edges at the plurality of distances using an image sensor to form a plurality of Bayer formats. After acquiring the image, the method may include calculating edge profile curves of the red channel, the green channel, and the blue channel for each of the plurality of Bayer format images. This embodiment obtains a plurality of Bayer format images at a target distance to which the depth of field is to be extended and at a distance other than the target distance, and then obtains an edge profile curve of each color channel for each Bayer format image. The blurring curve and the blurring function are obtained in a subsequent process for each of the edge profile curves obtained for each color channel, and the total blurring function is determined by summing and applying the weighting to the blurring functions obtained for each color channel. Done. In this embodiment, by calculating a blurring function at a plurality of distances and summing and applying the weights, a more efficient correction may be made by weighting the distances when the distances requiring correction for each color are different. In addition, the weighted summation method can be used to compensate for manufacturing tolerances occurring in the actual module.

Next, differentiating edge profile curves of the red, green and blue channels to obtain blurring curves of the red, green and blue channels (S32) and blur of the red, green and blue channels A step S33 for determining the blurring function of the red channel, the blurring function of the green channel, and the blurring function of the blue channel representing the ring curve will be described.

4 is a block diagram illustrating a process of obtaining a blurring function using a blurring curve after obtaining a blurring curve. In the present invention, in order to obtain a blurring function, first, a Bayer format image output from an image sensor must be directly acquired (S41). To this end, the ISP (14 of FIG. 1) located at the rear of the image sensor may be cut off to stop the function, and a Bayer format image output from the image sensor may be directly acquired by using a separate interface board and a capture program. In particular, in order to detect a blurring function, a black and white line edge is picked up by an image sensor and a Bayer format image output from the image sensor is directly acquired.

Subsequently, a red channel composed of red pixels, a green channel composed of green pixels, and a blue channel composed of blue pixels are separated from the Bayer format image output from the image sensor (42). The process of separating each of these color channels can be accomplished using a variety of commercial computer tools well known in the art. The Bayer format image output from the image sensor is an image composed of red pixels, green pixels, and blue pixels, as shown in FIG. 5. In the Bayer format image, as shown in FIG. 5, a green channel made of green pixels, a red channel made of red pixels, and a blue channel made of blue pixels are separated. As shown in FIG. 5, the green channel may include a first green channel including a green pixel adjacent to a red pixel in a horizontal direction, and a second green channel including a green pixel adjacent to a blue pixel in a horizontal direction although not shown. .

In each of the color channels shown in FIG. 5, the portion indicated by the dotted circle shows the spot size at the target distance. This spot size is not intended to show the spot size for a particular color channel, but to compare the number of pixels included in the spot in the Bayer format image output from the image sensor with the number of pixels included in the spot in each extracted color channel. . That is, as illustrated in FIG. 5, the sizes of the spots formed in the red channel, the green channel, and the blue channel included in one Bayer format image are all different, which causes a problem of deterioration of sharpness. Reducing this difference in spot size improves the sharpness of the image at the target distance, thereby extending the depth of field to ensure sharpness. When the horizontal spacing between each pixel of the Bayer format image output from the image sensor is Δx and the vertical spacing is Δy, the spacing between the pixels of each color channel is 2Δx, 2Δy. That is, each separated color channel may be understood to re-extract pixel values at intervals having a size twice the interval between pixels of the Bayer format image output from the image sensor.

Subsequently, the edge profile curves of the extracted red, green, and blue channels are obtained (43). To obtain an edge profile curve, a measurement area composed of arbitrary pixels including an edge is selected, and the edge profile curve is obtained in this measurement area. The edge profile curve for the image sensor output image that captures the ideal black and white line edge can be formed vertically at the edges of the black region and the white region. However, in the image sensor output image of the actual black and white line edges, blurring occurs at the edge portion, and the blurring profile causes the edge profile curve to be inclined around the edge portion as shown in FIG. Is formed. At this time, the number of pixels required to rise from 10% to 90% of the maximum value of the edge profile curve can be defined as a normal spot size. The greater the amount of blurring, the smoother the slope of the edge profile curve can be formed and the larger the spot size. This spot size is different for each color channel, and this difference causes a reduction in image sharpness as described above and a narrow depth of field.

In the present invention, the process of obtaining the edge profile curve of each color channel from the Bayer format image for the black and white line edges is preferably performed for both the horizontal black and white line edges and the vertical black and white line edges. The edge profile curve representing the blurring in the horizontal direction can be obtained, and the edge profile curve representing the blurring in the vertical direction can be obtained through the vertical black and white line edge.

Subsequently, the edge profile curves of the red, green, and blue channels are differentiated to obtain blurring curves of the red, green, and blue channels (44). Since the edge profile curve has a form in which the slope gradually increases and then the slope decreases again from the center of the edge, when the edge profile curve is differentiated, a Gaussian distribution curve as shown in FIG. Will form. This Gaussian distribution curve is defined as a blurring curve.

According to the present invention, a horizontal blurring curve may be obtained using an image of a vertical black and white line edge, and a vertical blurring curve may be obtained using an image of a horizontal black and white line edge.

The process of obtaining the above-described edge profile curve and the process of differentiating the edge profile curve to obtain the blurring curve may be performed using various commercial computer tools well known in the art.

Next, each one-dimensional blurring function representing the blurring curves for the red channel, the green channel, and the blue channel is determined (45). This blurring function determination process is a process of obtaining a mathematical expression represented by a Gaussian distribution curve as shown in (b) of FIG. 6 and can be obtained through a function fitting process that is commonly used. In this blurring function determination process, the horizontal blurring function may be determined from the horizontal blurring curve and the vertical blurring function may be determined from the vertical blurring curve. That is, this blurring function determination process can be understood as a 1-dimensional blurring function determination process for determining the blurring function in the horizontal direction and the vertical direction, respectively.

For example, the horizontal blurring function may be determined as in Equation 5 below.

[Equation 5]

(m _c -N _x ≤m≤m _c + N _x )

In Equation 5, Δx is the horizontal pixel spacing of the image sensor, A _x is the horizontal height of the blurring function, m _c is the horizontal center coordinate of the blurring function, and σ _x is the width of the horizontal blurring function. Indicates. In Equation 5, the height (A _x ) of the blurring function, the width (σ _x ) of the horizontal blurring function, and the horizontal center coordinate (m _c ) of the blurring function are fitting of the Gaussian distribution curve, which is a blurring curve. Can be obtained simply by In Equation 5, N _x represents an area for obtaining the blurring function, from the point representing 10% of the maximum value and the point representing 90% to the center pixel of the horizontal blurring curve in the edge profile curve. It can be a distance. In Equation 5, sigma _x may be defined as a blurring amount as a parameter related to the width of the blurring curve. In addition, the height A _x of the blurring curve in Equation 5 can be normalized so that the maximum value is 1 and does not have a significant meaning in the present invention for obtaining the blurring amount.

It can be seen from Equation 5 that the blurring function is determined in the 2N _x +1 pixel range in the horizontal direction, and data is sampled every two times the pixel interval of the Bayer format image output from the image sensor.

In addition, similar to the above equation 5, the vertical blurring function can also be determined.

Subsequently, a two-dimensional blurring function capable of representing both horizontal and vertical blurring amounts is obtained using the one-dimensional blurring function, that is, the horizontal and vertical blurring functions (46). In order to perform sharpness correction on an image detected by an actual camera module, it is not meaningful to apply only one 1D blurring function, and the 1D blurring function should be extended to a 2D blurring function. The two-dimensional blurring function that simultaneously represents the amount of horizontal and vertical blurring may be obtained by multiplying the horizontal blurring function and the vertical blurring function, which are the one-dimensional blurring function. This two-dimensional blurring function is represented by following formula (1).

[Equation 1]

(m _c -N _c ≤m≤m _x + N _x, n _{y c} -N ≤n≤n _c + N _y)

In Equation 1, Δx is the horizontal pixel spacing of the image sensor, Δy is the vertical pixel spacing of the image sensor, A is the height of the blurring function (the height of the horizontal blurring function and the height of the vertical blurring function M _c is the coordinate of the horizontal center pixel of the blurring function, n _c is the coordinate of the vertical center pixel of the blurring function, σ _x is the width of the horizontal blurring function, and σ _y is the vertical blurring. The width of the function.

A blurring function such as Equation 1 is calculated for the red channel, the green channel, and the blue channel, respectively.

The process of obtaining the blurring function described above corresponds to an embodiment using a Bayer format image obtained by capturing a black and white line edge at one target distance. In embodiments in which a plurality of Bayer format images are used by imaging line edges at a plurality of distances, the basic principle of obtaining an edge profile curve from which a blurring curve and a blurring function are obtained is the same as described above, Since the profile curves of the red, green, and blue channels are obtained for Bayer format images, the multiple edge profile curves, blurring curves, and blurring functions must be obtained for each color channel. The difference is that a total blurring function is obtained by weighting a plurality of blurring functions for.

In an embodiment using a plurality of Bayer format images, each blurring function representing the blurring curves for the plurality of red, green, and blue channels obtained for each of the plurality of Bayer format images is determined as in Equation 3 below. Can be.

[Equation 3]

(m _c -N _c ≤m≤m _x + N _x, n _{y c} -N ≤n≤n _c + N _y)

In Equation 3, b _p is a blurring function at the distance p, Δx is the horizontal pixel spacing of the image sensor, Δy is the vertical pixel spacing of the image sensor, A _p is the height of the blurring function at the distance p (horizontal Average of the height of the vertical blurring function and the height of the vertical blurring function), m _cp is the coordinate of the horizontal center pixel of the blurring function at distance p, and n _cp is the vertical center pixel of the blurring function at distance p. The coordinate, σ _xp, is the width of the horizontal blurring function at distance p, and σ _yp is the width of the vertical blurring function at distance p. When the black and white line edges are captured at each distance p, the positions of the edges are the same. Therefore, the horizontal and vertical center coordinates of the blurring function are the same at each distance p.

Equation 3 is a blurring function (two-dimensional) calculated for each color in one Bayer format image of the plurality of Bayer format images. The procedure for obtaining this blurring function is the same as the series of processes performed to obtain Equation 1 above.

The entire blurring function is calculated as shown in Equation 4 by using a plurality of blurring functions obtained as shown in Equation 3 for each color channel.

[Equation 4]

In Equation 4, b denotes the total blurring function, w _p denotes the weight, and the sum of the total weights multiplied by each of the plurality of individual blurring functions is 1.

For example, if w ₁ represents a weight at a short distance and w _N represents a weight at a distance, respectively, the short-range weight w ₁ has the largest value in the red channel and the green channel, as shown in FIG. In the case of the blue channel, the weight may be allocated such that the far weight w _n has the largest value. This is because the red channel and the green channel should improve the sharpness at a short distance inside the depth of field, while the blue channel should improve the sharpness at a long distance.

As described above, after determining the blurring functions (Equations 1 and 4) for the red channel, the green channel, and the blue channel, the present invention uses the blurring function as shown in FIG. And a correction coefficient for the blue channel (S341, S351, S361).

This correction coefficient may be defined in the form of an inverse filter in the frequency domain well known in the art as shown in Equation 2 below.

[Equation 2]

In Equation 2, D denotes a correction coefficient, B denotes a Fourier transform of the function b of Equation 1 or 4, Δu denotes the inverse of Δx, Δv denotes the inverse of Δy, and k denotes a parameter. .

The function 'D' expressed in Equation 2 represents a correction coefficient in the frequency domain, and 'B' is a Fourier transform in order to convert the blurring function, that is, Equation 1 or Equation 4 into the frequency domain. By the conversion to the frequency domain,? U and? V represent sampling intervals in the frequency domain, that is, intervals between pixels in the frequency domain. Specifically, Δu is equal to 1 / Δx, and Δv is equal to 1 / Δy. In addition, the parameter k in Equation 2 is for modifying the correction coefficient, so that its value can be appropriately determined to obtain an optimal correction coefficient. Specific methods for determining the value of the parameter k will be described separately below.

Next, the corrected red channel, green channel, and blue channel are generated by applying the correction coefficient determined by Equation 2 to the red, green, and blue channels of the Bayer format image detected from the image sensor (S342, S352). , S362). The function 'D' expressed in Equation 2 expresses a correction coefficient in the frequency domain, and in order to apply the correction coefficient to each pixel, the function 'D' is determined and then converted into a function of the spatial domain through an inverse Fourier transform. do. The correction coefficient transformed into the spatial domain has a form of a matrix filter, and the correction coefficient of the spatial domain is convolutionally calculated for each pixel of the Bayer format image output from the image sensor to correct the sharpness of each pixel. Will be performed.

On the other hand, in order to further improve the effect of the sharpness correction, it is necessary to determine the value of the correction coefficient (k) appropriately, for this purpose steps (S342, S352, S362) to the corrected red channel, green channel and blue channel The method may further include obtaining an edge profile for the edge profile and obtaining spot sizes of the red channel, the green channel, and the blue channel corrected using each edge profile.

Subsequently, the corrected spot sizes of the red, green, and blue channels are compared with predetermined target spot sizes, respectively (S343, S353, S363). The present invention improves the sharpness at the target distance by reducing the spot size of each color channel to the desired level to extend the depth of field to the desired level. Therefore, when the spot sizes of the corrected red, green and blue channels are not substantially the same as the target spot size, the value of the parameter k of the correction coefficient represented by Equation 2 is adjusted (S344, S354). , S364), this parameter (k) value is calculated and applied again to the red, green and blue channels of the Bayer format image. This iterative process may be performed until the spot size of the corrected red channel, green channel and blue channel image and the target spot size for each color channel are substantially the same.

On the other hand, ideally, the value of the parameter k of the correction coefficient is preferably determined so that the spot size of the red, green, and blue channels corrected by the correction coefficient and the target spot size for each color channel exactly match. However, since it is almost impossible to exactly match the two values in a practical application environment, the parameter (k) that minimizes the difference between the spot sizes of the corrected red, green and blue channels and the target spot size of each color channel. It is more preferable to determine the value of as a parameter of the final correction factor.

Hereinafter, a method of determining a correction coefficient, that is, a parameter k, to minimize the difference between the spot sizes of the corrected red, green and blue channels and the target spot size of each color channel will be described in detail. do.

7 is a flowchart illustrating a method of determining a correction coefficient such that the difference between the spot size and the target spot size of each corrected color channel is minimized.

Referring to FIG. 7, first, a correction coefficient for each color channel is determined as in Equation 2 (S71). At this time, the initial value of the parameter k can be appropriately determined by experimental experience. In addition, in the process of determining the appropriate correction coefficient by changing the parameter afterwards, the correction value for modifying the parameter (k) and its modification range may also be preset based on experimental experience.

Subsequently, each color channel is corrected by applying a correction factor (S72). In the step of calculating the correction coefficient (S61), the initial value of the parameter of the correction coefficient ('k' in Equation 2) may be appropriately determined experimentally. Correcting each color channel by applying the correction coefficient (S62) has already been described above, so a detailed description thereof will be omitted.

Subsequently, a horizontal edge profile curve is obtained from each of the corrected color channels, and a horizontal spot size of the corrected color channel is obtained from the horizontal edge profile curve (S731). Similarly, a vertical edge profile curve is obtained from each of the corrected color channels, and a vertical spot size of the corrected color channel is obtained from the vertical edge profile curve (S741). The technique for obtaining the spot size is also as described above.

Subsequently, the horizontal score of the corrected color channel is calculated using the target spot size of the corresponding color channel and the horizontal spot size of the corrected color channel (S732), and the target spot size and the corrected color channel of the corresponding color channel are calculated. A vertical score for the corrected color channel is calculated by using the vertical spot size of (S742). The horizontal score may be defined as a difference between the target spot size of the color channel to be compared with the horizontal spot size of the corrected color channel. Similarly, the vertical score is the target spot of the color channel to be compared. It can be defined as the difference between the size and the vertical spot size of the corrected color channel.

Subsequently, the total score for the corrected color channel is calculated by summing the horizontal score and the vertical score for the corrected color channel and storing the value (S75).

Subsequently, the value of the parameter k is adjusted by the initially set correction value (S76), and it is determined whether the value of the adjusted parameter k is within the initially set correction range (S77), and within the correction range. If the above is repeated for the above-described score calculation and storage process, if it is out of the correction range, the process of ending the score calculation process, after modifying the value of the parameter (k) within the correction range, the total score of the total scores calculated and stored Is finally determined as a parameter of the correction coefficient (S78). This makes it possible to determine the correction coefficient having the smallest difference between the spot size and the target spot size of the color channel corrected in the horizontal and vertical directions.

FIG. 8 is a diagram illustrating a spot size of each color channel of a Bayer format image to which an image processing method for expanding depth of field according to the present invention is applied.

As shown in Fig. 8, the target spot size is determined as the spot size in a range (target distance) to extend the depth of field. As shown in FIG. 8, in the present invention, the target spot size is less than 8 pixels and less than 4 pixels in the target distance for the red channel Rc and the green channel Gc, respectively, and less than 4 pixels in the distance in the blue channel Bc. Calibration was done by setting In the present invention, the target spot size at the target distance of the red and green channel images is set to be the same as the spot size of the red and green channel images at 50 cm, which is a near boundary of the depth of field. In the conventional camera module, in the blue channel, resolution does not have a big problem in a range to extend the depth of field, but a problem occurs at a long distance. When the correction coefficient calculated by the above method is calculated to satisfy the target spot size at the target distance, it can be confirmed through FIG. 8 that the depth of field of the camera module can be extended from 50 cm to the target distance.

On the other hand, since there is a limit to the amount of blurring of an image that can be corrected using a correction factor, it is more preferable to determine the extended target distance of the depth of field according to the sharpness correction capability of the image processing technique and the amount of image blurring at a short distance of the optical system. .

As described above, according to the present invention, the depth of field can be extended without adding a separate physical configuration to the existing camera module. That is, the present invention has the advantage of eliminating the inconvenience of lens manufacturing problems that may occur by designing a special dedicated optical system, the sensitivity problems occurring when assembling the lens, the inconvenience of searching for a focusing position generated when assembling the camera module.

In addition, according to the present invention, the sharpness correction filter coefficients are calculated in advance in consideration of the degradation of the resolution of the Bayer image by the point diffusion function of the optical system, and then the sharpness of the Bayer image is corrected using the camera module, even if the optical system configuration is changed. After recalculating only the correction factor suitable for, it can be applied to depth of field. Therefore, the camera can extend the depth of field of the camera module by applying the same image processing method to the different camera modules without additional program modification.

Claims (14)

a) placing a black and white line edge at a target distance to extend the depth of field, and imaging the black and white line edge using an image sensor to obtain a Bayer format image; b) obtaining edge profile curves of the red, green and blue channels of the Bayer format image; c) differentiating edge profile curves of the red, green, and blue channels to obtain blurring curves of the red, green, and blue channels; d) determining the blurring function of the red channel, the blurring function of the green channel, and the blurring function of the blue channel, representing the blurring curves of the red channel, the green channel, and the blue channel; Function is a Gaussian distribution function; e) a correction coefficient for the red channel, a correction coefficient for the green channel, and a correction coefficient for the red channel for correcting the blurring present in the red channel, the green channel, and the blue channel, respectively, by using the blurring functions of the red, green, and blue channels; Obtaining a correction coefficient for the blue channel; And f) applying a correction coefficient for the red channel, a correction coefficient for the green channel, and a correction coefficient for the blue channel to the red, green, and blue channels of the Bayer format image detected by the image sensor, respectively. Image processing method for depth of field comprising a. The method of claim 1, The correction coefficient for the red channel, the correction coefficient for the green channel, and the correction coefficient for the blue channel have parameters for adjusting the blur correction amount for each color channel. Way. The method of claim 2, g) an edge profile curve of the red channel, the green channel, and the blue channel is obtained by applying a correction coefficient for the red channel, a correction coefficient for the green channel, and a correction coefficient for the blue channel, respectively, and the corrected red color. Obtaining a spot size of the corrected red channel, a spot size of the corrected green channel, and a spot size of the corrected blue channel from the edge profile curves of the channel, the green channel, and the blue channel; h) the spot size of the corrected red channel, the spot size of the corrected green channel, the spot size of the corrected blue channel and the target spot size of the preset red channel, the target spot size of the preset green channel, and the preset blue Comparing the target spot sizes of the channels, respectively; And i) calculating the corrected correction coefficient for the red channel by adjusting the value of the parameter of the correction coefficient for the red channel until the spot size of the corrected red channel and the target spot size of the red channel are substantially equal. And adjust the value of the parameter of the correction coefficient for the green channel until the spot size of the corrected green channel and the target spot size of the green channel are substantially the same. Calculate the adjusted correction coefficient for the blue channel by adjusting a value of a parameter of the correction coefficient for the blue channel until the spot size of the corrected blue channel and the target spot size of the blue channel are substantially the same. And calculating the depth of field. The method according to any one of claims 1 to 3, Each blurring function representing the blurring curves for the red channel, the green channel, and the blue channel is as in Equation 1 below. [Equation 1]

(m _c -N _c ≤m≤m _x + N _x, n _{y c} -N ≤n≤n _c + N _y) (Δx: horizontal pixel spacing of the image sensor, Δy: vertical pixel spacing of the image sensor, A: height of the blurring function (average of the height of the horizontal blurring function and the height of the vertical blurring function), m _c : Coordinate of the horizontal center pixel of the blurring function, n _c : Coordinate of the vertical center pixel of the blurring function, σ _x : Width of the horizontal blurring function, σ _y : Width of the vertical blurring function) The method of claim 4, wherein The correction coefficient is an image processing method for expanding the depth of field, characterized in that as shown in Equation 2. [Equation 2]

(D: correction coefficient, B: Fourier transform of function b of Equation 1, Δu = 1 / Δx, Δv = 1 / Δy, k: parameters) The method of claim 2, j) the spot sizes of the red, green, and blue channels corrected in step f), the target spot size of the preset red channel, the target spot size of the preset green channel, and the target spot size of the preset blue channel, respectively. Calculating and storing the compared scores; k) modify the correction coefficient by changing the parameter, correct the red channel, green channel and blue channel of the Bayer format image detected by the image sensor by applying the corrected correction coefficient, and then correct the red Computing and storing scores comparing the spot sizes of the channels, the green channels and the blue channels, the target spot sizes of the red channels, the target spot sizes of the green channels, and the target spot sizes of the blue channels, respectively-this step Is performed repeatedly by changing the parameter; And l) determining a correction coefficient having a parameter for which a score having a minimum value among the stored scores is calculated as a final correction coefficient. The method of claim 6, Step j) and k), The horizontal score of the red channel is compared with the corrected horizontal spot sizes of the red, green and blue channels, the target spot size of the red channel, the target spot size of the green channel, and the target spot size of the blue channel, respectively. Calculating a horizontal score of the channel and a horizontal score of the blue channel; The vertical score of the red channel is compared with the corrected vertical spot sizes of the red, green and blue channels, the target spot size of the red channel, the target spot size of the green channel, and the target spot size of the blue channel, and green. Calculating a vertical score of the channel and a vertical score of the blue channel; And Calculating a sum of the horizontal scores of the red, green, and blue channels and the vertical scores of the red, green, and blue channels, and storing the sums of the red, green, and blue channels as total scores for the red, green, and blue channels, respectively. Image processing method for extending the depth of field comprising a. a) positioning a black and white line edge at a target distance to extend the depth of field and at least one arbitrary distance between the target distance and the distance in the depth of field, and using an image sensor to place the black and white line edges at the plurality of distances; Photographing the image to obtain a plurality of Bayer format images; b) obtaining edge profile curves of a red channel, a green channel, and a blue channel of each of the plurality of Bayer format images; c) differentiate the edge profile curves of each of the red, green, and blue channels of each of the plurality of Bayer format images to obtain a plurality of blurring curves for the red channel, a plurality of blurring curves for the green channel, and a blue channel. Obtaining a plurality of blurring curves for; d) determine a blurring function for a plurality of red channels representing the blurring curves for the plurality of red channels, and a blurring function for the plurality of green channels representing the blurring curves for the plurality of green channels Determining a blurring function for the plurality of blue channels representing the blurring curves for the plurality of blue channels, wherein the plurality of blurring functions for each color channel are Gaussian distribution functions; e) apply a weight to each of the blurring functions for the plurality of red channels and add them all to obtain a total blurring function for the red channel, and apply a weight to each of the blurring functions for the plurality of green channels and add them all together Obtaining a total blurring function for the green channel, applying a weight to each of the blurring functions for the plurality of blue channels, and summing all of them to obtain a total blurring function for the blue channel; f) a correction coefficient for the red channel, for the green channel, respectively for correcting the blurring present in the red channel, the green channel, and the blue channel by using the total blurring functions for the red channel, the green channel, and the blue channel, respectively; Obtaining a correction coefficient for the correction coefficient and the blue channel; And g) applying a correction coefficient for the red channel, a correction coefficient for the green channel, and a correction coefficient for the blue channel to the red, green, and blue channels of the Bayer format image detected by the image sensor, respectively. Image processing method for depth of field comprising a. The method of claim 8, The correction coefficient for the red channel, the correction coefficient for the green channel, and the correction coefficient for the blue channel have parameters for adjusting the blur correction amount for each color channel. Way. The method of claim 9, h) an edge profile curve of the red channel, the green channel, and the blue channel is obtained by applying a correction coefficient for the red channel, a correction coefficient for the green channel, and a correction coefficient for the blue channel, respectively, and the corrected red color. Obtaining a spot size of the corrected red channel, a spot size of the corrected green channel, and a spot size of the corrected blue channel from the edge profile curves of the channel, the green channel, and the blue channel; i) the spot size of the corrected red channel, the spot size of the corrected green channel, the spot size of the corrected blue channel and the target spot size of the preset red channel, the target spot size of the preset green channel, and the preset blue Comparing the target spot sizes of the channels, respectively; And j) calculating a corrected correction coefficient for the red channel by adjusting a value of a parameter of the correction coefficient for the red channel until the spot size of the corrected red channel and the target spot size of the red channel are substantially equal. Calculating a corrected correction coefficient for the green channel by adjusting a value of a parameter of the correction coefficient for the green channel until the spot size of the corrected green channel and the target spot size of the green channel are substantially the same. Calculating a corrected correction coefficient for the blue channel by adjusting a value of a parameter of the correction coefficient for the blue channel until the spot size of the corrected blue channel and the target spot size of the blue channel are substantially the same. The image processing method for the depth of field further comprising the step of. The method according to any one of claims 8 to 10, Each of the blurring functions representing the blurring curves for the plurality of red, green, and blue channels is represented by Equation 3 below. The total blurring curves for the red channel, the green channel, and the blue channel are as shown in Equation 4 below. [Equation 3]

(m _c -N _c ≤m≤m _x + N _x, n _{y c} -N ≤n≤n _c + N _y) (b _p : blurring function at distance p, Δx: horizontal pixel spacing of image sensor, Δy: vertical pixel spacing of image sensor, A _p : height of blurring function at distance p (horizontal blurring function) Is the mean of the height of the vertical blurring function, and m _c is the coordinate of the horizontal center pixel of the blurring function at distance p (same for all distances), n _c is the vertical direction of the blurring function at distance p Coordinates of the center pixel (same at all distances), σ _xp : width of horizontal blurring function at distance p, σ _yp : width of vertical blurring function at distance p) [Equation 4]

(b: full blurring function, w _p : weight) The method of claim 11, The correction coefficient is an image processing method for expanding the depth of field, characterized in that as shown in Equation 2. [Equation 2]

(D: correction coefficient, B: Fourier transform of function b of Equation 4, Δu = 1 / Δx, Δv = 1 / Δy, k: parameter) The method of claim 9, k) the spot sizes of the red, green and blue channels corrected in step g), the target spot size of the preset red channel, the target spot size of the preset green channel, and the target spot size of the preset blue channel, respectively. Calculating and storing the compared scores; l) modifying the correction coefficient by changing the parameter, correcting the red channel, green channel and blue channel of the Bayer format image detected by the image sensor by applying the corrected correction coefficient, and then correcting red Computing and storing scores comparing the spot sizes of the channels, the green channels and the blue channels, the target spot sizes of the red channels, the target spot sizes of the green channels, and the target spot sizes of the blue channels, respectively-this step Is performed repeatedly by changing the parameter; And m) determining a correction coefficient having a parameter for which a score having a minimum value among the stored scores is calculated as a final correction coefficient. The method of claim 13, Step k) and step l), The horizontal score of the red channel is compared with the corrected horizontal spot sizes of the red, green and blue channels, the target spot size of the red channel, the target spot size of the green channel, and the target spot size of the blue channel, respectively. Calculating a horizontal score of the channel and a horizontal score of the blue channel; The vertical score of the red channel is compared with the corrected vertical spot sizes of the red, green and blue channels, the target spot size of the red channel, the target spot size of the green channel, and the target spot size of the blue channel, and green. Calculating a vertical score of the channel and a vertical score of the blue channel; And Calculating a sum of the horizontal scores of the red, green, and blue channels and the vertical scores of the red, green, and blue channels, and storing the sums of the red, green, and blue channels as total scores for the red, green, and blue channels, respectively. Image processing method for extending the depth of field comprising a.

Images