KR20210106867A - Method and apparatus for calibrating an image sensor - Google Patents

Abstract

A method and apparatus for calibrating an image sensor are disclosed. According to an embodiment of the present disclosure, a method of calibrating an image sensor includes the steps of: acquiring a plurality of multi-light source images generated from a plurality of image sensor modules; calculating a plurality of crosstalk levels and a plurality of color-specific correction coefficients based on the plurality of multi-light source images; generating modeling data based on a relationship between a crosstalk level for a first color and the plurality of color-specific correction coefficients; and calculating the color correction coefficient for the target pixel of the image sensor provided in the first image sensor module based on a single light source image captured by a first image sensor module and the modeling data.

Description

Method and apparatus for calibrating an image sensor

The technical idea of the present disclosure relates to calibration of an image sensor, and more particularly, to a method and apparatus for calibrating an image sensor for extracting a correction coefficient for correcting crosstalk in the image sensor.

The image sensor may include a pixel array for sensing a received light signal. In order to obtain high-quality images, the pixels of the image sensor are highly integrated. Further, as the image sensor provides an auto-focusing function, the pixel array may include phase detection pixels, and the phase detection pixels may be discretely and regularly disposed within the pixel array. As the pixel size in the image sensor decreases and the pattern of the pixel array becomes non-uniform, there is a problem in that crosstalk between pixels increases. Crosstalk means that a signal generated from an adjacent pixel affects an arbitrary pixel, changes spectral characteristics of a signal generated in an arbitrary pixel, and deteriorates color reproducibility. Accordingly, the image quality may be deteriorated. Therefore, crosstalk correction that adjusts pixel values so that the influence of crosstalk is removed or minimized is required.

An aspect of the present disclosure is to provide a calibration method and apparatus for an image sensor for efficiently extracting a correction coefficient for correcting crosstalk of the image sensor.

According to an embodiment of the present disclosure for achieving the above technical problem, a calibration method of an image sensor includes: acquiring a plurality of multi-light source images generated from a plurality of image sensor modules; based on the plurality of multi-light source images, a plurality of calculating a crosstalk level and a plurality of color-specific correction coefficients of , generating modeling data based on a relationship between a crosstalk level for a first color and the plurality of color-specific correction coefficients, and a first image sensor The method may include calculating the correction coefficient for each color of the target pixel of the image sensor provided in the first image sensor module based on the single light source image captured by the module and the modeling data.

According to an embodiment of the present disclosure for achieving the above technical problem, an apparatus for calibrating an image sensor includes a memory for storing modeling data representing color correction coefficient functions for a target pixel and a single light source image obtained from the image sensor and a processor that calculates the crosstalk level of the target pixel from the data and calculates a correction coefficient for each color of the target pixel based on the modeling data and the crosstalk level of the target pixel.

According to an embodiment of the present disclosure for achieving the above technical problem, a method of calibrating an image sensor includes: calculating a crosstalk level of a target pixel on which crosstalk correction is to be performed from a single light source image obtained from the image sensor; and calculating a correction coefficient for each color of the target pixel by applying the crosstalk level to color correction coefficient functions calculated in advance for the target pixel.

According to the method and apparatus for calibrating an image sensor according to the technical concept of the present disclosure, modeling data indicating the relationship between the crosstalk level and the correction coefficients for each color is generated and stored in advance based on a multi-light source image, and the image sensor module is calibrated In the step, the correction coefficients for each color may be calculated based on the single light source image and the modeling data. Accordingly, in the calibration step of the image sensor module, additional equipment necessary for acquiring a multi-light source image, for example, installation of a plurality of light sources is not required, and since the correction coefficient for each color is calculated based on the single light source image, processing burden and processing time are reduced. can be reduced. In addition, since the image sensor may perform crosstalk correction based on correction coefficients for each color, appropriate correction may be performed on an image obtained by capturing a subject including various colors.

In order to more fully understand the drawings recited in the Detailed Description of the present disclosure, a brief description of each drawing is provided.
1 is a block diagram illustrating an image sensor and an electronic device including the same according to an exemplary embodiment of the present disclosure.
2A and 2B exemplarily show a pixel array.
3A schematically illustrates a vertical cross-sectional view of a normal pixel, and FIG. 3B schematically illustrates a vertical cross-sectional view of a phase difference detection pixel.
4 is a flowchart illustrating a calibration method of an image sensor according to an exemplary embodiment of the present disclosure.
5 shows an example in which a pixel array is divided into a plurality of grids.
6A, 6B, and 6C are graphs showing the relationship between the crosstalk level and the correction coefficient.
7 illustrates an example of an image sensor calibration apparatus according to an exemplary embodiment of the present disclosure.
8 shows modeling data according to an exemplary embodiment of the present disclosure.
9 shows an example of an apparatus for calibrating an image sensor according to an exemplary embodiment of the present disclosure.
10 is a block diagram schematically illustrating an electronic system according to an exemplary embodiment of the present disclosure.

Hereinafter, various embodiments of the present disclosure are described in connection with the accompanying drawings.

1 is a block diagram illustrating an image sensor and an electronic device including the same according to an exemplary embodiment of the present disclosure.

The image sensor 100 may convert an optical signal of an object incident through the optical lens LS into image data. The image sensor 100 may be mounted on an electronic device having an image or light sensing function. For example, the image sensor 100 may include a digital still camera, a digital video camera, a smartphone, a wearable device, an Internet of Things (IoT) device, a tablet PC (Personal Computer), a PDA (Personal Digital Assistant), It may be mounted on the electronic device 1000 such as a portable multimedia player (PMP), a navigation device, or the like. In addition, the image sensor 100 may be mounted on the electronic device 1000 provided as a component in a vehicle, furniture, manufacturing equipment, door, various measurement devices, and the like. For example, the electronic device 1000 may be implemented as a mobile device such as a smart phone or a wearable device, and the processor 200 may be an application processor (AP).

Referring to FIG. 1 , the image sensor 100 may include a pixel array 110 , a readout circuit 120 , and an image signal processor 130 . In an embodiment, the pixel array 110 , the readout circuit 120 , and the image signal processor 130 may be implemented as a single semiconductor chip or semiconductor module. In an embodiment, the pixel array 110 and the readout circuit 120 may be implemented as one semiconductor chip or semiconductor module, and the image signal processor 130 may be implemented as another semiconductor chip or semiconductor module.

The pixel array 110 may be implemented as, for example, a photoelectric conversion device such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and may be implemented with various types of photoelectric conversion devices. The pixel array 110 includes a plurality of pixels that convert a received optical signal (light) into an electrical signal, and the plurality of pixels may be arranged in a matrix.

Each of the plurality of pixels includes a photo-sensing element (or referred to as a photoelectric conversion element). For example, the photosensitive device may include a photodiode, a phototransistor, a port gate, a pinned photodiode, a perovskite photodiode, and an organic photodiode, an organic photoconductive film, and the like. In addition, various photo-sensing devices may be applied.

The plurality of pixels may sense light using a photo-sensing device and convert the sensed light into an electrical signal. Each of the plurality of pixels may detect light in a specific spectral region. For example, the plurality of pixels include a pixel for converting light in a red spectrum region into an electric signal (hereinafter, referred to as a red pixel), and a pixel for converting light in a green spectrum region into an electric signal (hereinafter referred to as a green pixel). ), and a pixel (hereinafter, referred to as a blue pixel) that converts light in a blue spectrum region into an electrical signal. However, the present invention is not limited thereto, and the plurality of pixels may further include a white pixel. As another example, the plurality of pixels may include pixels combined in different color configurations, for example, a yellow pixel, a cyan pixel, and a green pixel.

A color filter array for transmitting light in a specific spectral region may be disposed above the plurality of pixels, and a color detectable by the corresponding pixel may be determined according to the color filter disposed above each of the plurality of pixels. However, the present invention is not limited thereto, and in the case of a specific photo-sensing device, light of a specific wavelength band may be converted into an electrical signal according to a level of an electrical signal applied to the photo-sensing device.

2A and 2B exemplarily show a pixel array.

Referring to FIG. 2A , the pixel array 110a may include color filters in which nine pixels PX arranged in 3 Х 3 columns have the same color. 9 red pixels (R), 9 green pixels (Gr), 9 blue pixels (B), and 9 green pixels (Gb) each arranged in 3 Х 3 rows (hereinafter, green pixels (Gr) are the first A pixel group PG consisting of one green pixel and a green pixel Gb will be referred to as a second green pixel) has a structure in which the pixel group PG is repeatedly arranged. The pixel array 110b having this structure may be referred to as a Nona CFA.

Even in pixels of the same color, a signal difference may occur between signals generated by the pixels PX according to pixels disposed around the pixels. For example, both the first pixel PX1 and the second pixel PX2 are red pixels R, and may be disposed at similar positions in the pixel array 110a. However, the first green pixel Gr and the red pixel R are mainly disposed near the first pixel PX1 , and the first pixel PX1 has the first green pixel Gr and the red pixel R near the first pixel PX1 . will be affected by In addition to the first green pixel Gr and the red pixel R, the second green pixel Gb and the blue pixel B are disposed near the second pixel PX2 . Accordingly, the second pixel PX2 is affected by the surrounding first green pixel Gr, red pixel R, second green pixel Gb, and blue pixel B.

Meanwhile, the pixel array 110a may have a CFA structure in which pixels of the same color are arranged in k Х k matrix (k is an integer of 2 or more), in addition to the Nona CFA as shown in FIG. 2B .

Referring to FIG. 2B , in the pixel array 110b , some pixel groups PG among the repeatedly arranged pixel groups PG may include the phase difference detection pixel PX _PD . In an embodiment, the plurality of phase difference detection pixels PX _PD may be sequentially arranged in a row or column.

The pixel array 110b may include a red pixel, a green pixel, and a blue pixel, and four pixels PX arranged in a 2 Х 2 matrix may include a color filter of the same color. A pixel group (PG) consisting of 4 red pixels (R), 4 green pixels (Gr), 4 blue pixels (B) and 4 green pixels (Gb), each arranged in a 2 Х 2 matrix, is repeatedly formed It has an arranged structure. The pixel array 110a having such a structure may be referred to as a quad Bayer color filter array (CFA). However, the present invention is not limited thereto, and the pixel array 110b may have a Bayer CFA structure including a red pixel, two green pixels, and a blue pixel in which the pixel groups PG are arranged in 2 Х 2 rows, or the same color. It may have a CFA structure in which pixels of k Х k are arranged in an integer of 2 or more.

_{The phase difference detection pixel PX PD} will be described with reference to FIGS. 3A and 3B .

3A schematically illustrates a vertical cross-sectional view of a normal pixel, and FIG. 3B schematically illustrates a vertical cross-sectional view of a phase difference detection pixel.

Referring to FIG. 3A , each of the plurality of pixels PX may include a photo-sensing element, for example, a photodiode PD and a color filter CF, and a micro lens ML may be provided on the color filter CF. have. The micro lens ML may be circular.

Referring to FIG. 3B , the phase detection pixel PX _PD includes a plurality of sub-pixels SPX having the same size as the pixel PX of FIG. 3A , and one micro lens is disposed on the plurality of sub-pixels SPX. (ML) may be placed. For example, when the phase detection pixel PX _PD includes two sub-pixels SPX, the micro lens ML may have an elliptical shape. As such, the structure of the phase detection pixel PX _PD may be different from a general pixel (eg, the pixel PX of FIG. 3A ).

Continuing to refer to FIGS. 2A and 2B , in a pixel array having a regular pattern, crosstalk between pixels PX is similar, and thus visibility of crosstalk may be weak. However, as shown in FIGS. 2A and 2B , when the pattern of the pixel arrays 110a and 110b is irregular, that is, when neighboring pixels of the pixels PX of the same color are arranged differently, the pixels PXs Since the crosstalk is different between the two, the visibility of the crosstalk may be strong. For example, crosstalk may be different between pixels PX of the same color due to a difference in color between adjacent pixels, _{a microlens shape of the phase detection pixel PX PD, and the like, so that a signal difference may occur.} The image quality may be degraded. Therefore, crosstalk correction that adjusts pixel values so that the influence of crosstalk is removed or minimized is required.

Continuing to refer to FIG. 1 , the readout circuit 120 receives electrical signals from the pixel array 110 and converts them into digital data, thereby image data including pixel values corresponding to each of a plurality of pixels. (also called an image) can be created. Image data is raw raw data.

The image signal processor 130 may perform image processing on the image data IDR1 output from the readout circuit 120 . For example, the image signal processor 130 may perform image processing such as bad pixel correction, crosstalk correction, and noise removal on image data, for example, the first image data IDT1 .

The image signal processor 130 may include a color correcting unit 131 , and the color correcting unit 131 is a pixel, for example, a correction for removing crosstalk from a pixel value for each of the pixels (PX in FIG. 2 ). (hereinafter referred to as crosstalk correction) may be performed.

The correction coefficients Cr, Cg, and Cb set for each color of the pixels PX requiring crosstalk correction, for example, the red correction coefficient (Cr), the green correction coefficient (Cg), and the blue correction coefficient (Cb) are It may be stored in the memory 132 , and the color correction unit 131 may perform crosstalk correction based on the correction coefficients Cr, Cg, and Cb.

The correction coefficients Cr, Cg, and Cb are calculated based on the crosstalk level of the single light source image data generated by the image sensor 100 imaging a single light source (eg, white) in the calibration process of the image sensor 100 . can be A function for each correction coefficient may be generated based on a plurality of multi-light source image data captured by a plurality of image sensors (eg, images of various primary colors generated by imaging light sources of various primary colors), and the function includes the image sensor 100 ) by applying the crosstalk level of the generated single light source image data, the correction coefficients Cr, Cg, and Cb for each color of the image sensor 100 may be calculated from the function. A method of calibrating the image sensor 100 according to an exemplary embodiment of the present disclosure will be described later in detail with reference to FIG. 4 .

In FIG. 1 , the memory 132 is illustrated as being provided inside the image signal processor 130 , but is not limited thereto, and the memory 132 is disposed in the image signal processor 130 or implemented as a separate semiconductor chip. can be

In an embodiment, as described with reference to FIG. 2A , when crosstalk is different between pixels of the same color within a pixel group, the color corrector 131 may perform crosstalk correction on the pixels. .

In an embodiment, the color correction unit 131 may include correction coefficients Cr set for each of the pixels PX disposed in the vicinity of the _{phase detection pixel PX PD, as described with reference to FIG. 2B .} Crosstalk correction by the _{phase detection pixel PX PD} may be performed based on Cg and Cb.

The image signal processor 130 may further include a plurality of image processing IPs (Intellectual Property) for performing the above-described image processing in addition to the color corrector 131 , and image-processed image data, for example, second image data (IDT2) may be provided to the processor 200 (eg, an application processor, a main processor of the electronic device 1000, or a graphic processor).

4 is a flowchart illustrating a calibration method of an image sensor according to an exemplary embodiment of the present disclosure.

With the calibration method of FIG. 4 , correction coefficients for each color for correcting crosstalk of the image sensor may be extracted. For example, a calibration method for extracting a correction coefficient for each color for correcting crosstalk by a phase detection pixel will be described.

In the calibration method according to an exemplary embodiment of the present disclosure, in the calibration step of the image sensor module (or camera module) including the image sensor ( 100 in FIG. 1 ), a calibration device, for example, a set-up device (or set-up device) ) can be done by In this case, the image sensor module may include an image sensor ( 100 in FIG. 1 ), an optical lens ( LS in FIG. 1 ), and a housing that forms the exterior of the image sensor module, and also includes a filter (eg, an infrared cut filter). ), an actuator, and a driving circuit for driving the actuator may be further included.

A plurality of image sensor modules and a specific image sensor module mentioned in this embodiment may have the same configuration and physical characteristics of the configurations. For example, the pixel array structure of the image sensor 100 provided in the image sensor module, the arrangement of pixels, and the structure and number of phase detection pixels may be the same, and the angle of view of the optical lens LS may be the same between the image sensor modules.

Referring to FIG. 4 , a calibration method according to an exemplary embodiment of the present disclosure may include a multi-light source calibration step S10 and a single light source (single light source) calibration step S20 . The multi-light source calibration step S10 may be performed using a plurality of image sensor modules. In an embodiment, the multi-light source calibration step ( S10 ) is performed before the mass-production step of the image sensor module, and various parameter setting steps of a specific image sensor module in the aspect step of the image sensor module (eg, during setup of the calibration process) ), the single light source calibration step S20 may be performed.

A plurality of multi-light source images may be obtained from each of the plurality of image sensor modules ( S11 ). Here, the multi-light source image refers to a plurality of primary color images generated by the image sensor module capturing a plurality of primary color light sources of different colors, respectively. In order to calculate correction coefficients for each color, for example, correction coefficients cr, Cg, and Cb for red, green, and blue colors, a primary color image of at least three colors or more is required. Each of the plurality of image sensor modules may generate images of red, green, and blue, that is, a multi-light source image, by imaging light sources corresponding to a plurality of colors, for example, red, green, and blue light sources. In an embodiment, the plurality of image sensor modules may further generate a white image by capturing a white light source.

A plurality of crosstalk levels and a plurality of color correction coefficients may be calculated based on the plurality of multi-light source images ( S12 ). A crosstalk level and a color for each of the target pixels (pixels to which crosstalk correction is to be performed, for example, pixels disposed in the periphery of the phase detection pixel) based on the plurality of multi-light source images generated in step S11 Star correction coefficients (eg Cr, Cg, Cb) may be calculated.

The crosstalk XT may be expressed by Equation (1).

Here, Cr, Cg, and Cb represent a red correction coefficient, a green correction coefficient, and a blue correction coefficient, respectively, and Dr, Dg, and Db are pixel values of a target pixel in the captured image for each of red, green, and blue, cross The torque may be calculated based on pixel values of pixels surrounding the target pixel for which the torque is calculated.

Crosstalks XT1, XT2, and XT3 of the target pixel may be calculated from each of the multi-light source images, for example, red, green, and blue images generated by one image sensor module capturing the multi-light sources. For example, in each of the multi-light source images, crosstalks XT1 , XT2 , and XT3 of a target pixel may be calculated through comparison between pixel values. The crosstalks XT1, XT2, and XT3 may be expressed by Equation (2).

Equation 2 can be converted into Equation 3.

Compensation coefficients Cr, Cg, and Cb for each color may be calculated according to Equation (3). Crosstalk level and color-specific correction coefficients may be calculated for each of the multi-light source images. Accordingly, a plurality of cross-talk levels and a plurality of color-specific correction coefficients may be calculated from a plurality of multi-light source images obtained from a plurality of image sensors. can

As shown in Equations 2 and 3, crosstalk (XT1, XT2, XT3) of pixels in an image of at least three colors is required to calculate color-specific coefficients for three colors.

In an embodiment, in step S12 , a crosstalk level for the white image may be calculated. For example, a white image may be generated based on the red, green, and blue images, and a crosstalk level in the white image may be calculated. However, the present invention is not limited thereto, and as the image sensor captures the white light source, a white image may be obtained, and a crosstalk level of pixels may be calculated from the white image.

In an embodiment, the pixel array ( 110 of FIG. 1 ) may be divided into a plurality of grids, and for each of the plurality of grids, a crosstalk level for each target pixel and correction coefficients for each color are calculated. can

5 shows an example in which a pixel array is divided into a plurality of grids.

Referring to FIG. 5 , the pixel array 110 may include nine grids GRD, and each of the plurality of grids GRD includes a plurality of pixel groups PG and has the same pattern, for example, an arrangement of pixels. can As described with reference to FIGS. 2A and 2B , each of the plurality of pixel groups PG may include a plurality of color pixels, for example, a red pixel, a green pixel, and a blue pixel. In an embodiment, at least one of the plurality of pixel groups PG included in one grid GRD may include a phase detection pixel (PX _PD of FIG. 3A ). In each of the plurality of grids GRD, the phase detection pixels (PX _PD of FIG. 3A ) may be located at the same point.

As such, the plurality of grids GRD may have the same structure. However, crosstalk may be different between pixels disposed at the same point in each of the plurality of grids GRID. For example, a pixel to be corrected (referred to as a first pixel and a second pixel, respectively) of the same color is located at the same point P1 in the first grid GRID and the second grid GRD2, and the perimeter of the first pixel and the second pixel Even if the types and colors of the pixels located in the , crosstalk of the first pixel of the first grid GRID and the second pixel of the second grid GRD2 are different, crosstalk may be different. For example, dispersion induced during the manufacturing process of the image sensor 100 or within the pixel array 110 , the first grid GRID and the second grid GRD2 are positioned according to the positions of the first grid GRID and the second grid GRD2 . Since the amount of light received by the second grid GRID is different, characteristics of the first grid GRID and the second grid GRD2 may be different, and accordingly, crosstalk between the first pixel and the second pixel may be different. can Accordingly, when correcting the crosstalk for the first pixel and the second pixel, different correction coefficients for each color may be applied. Therefore, in operation S12 , a correction coefficient for each color of the target pixel may be calculated for each of the plurality of grids GRD.

Meanwhile, although it has been described that the correction coefficients Cr, Cg, and Cb for each of the red, green, and blue colors are calculated, the present invention is not limited thereto, and as described above, the pixel array 110 corresponds to a combination of different colors. is composed of pixels, and correction coefficients for other colors may be calculated. As an example, the pixel array may include a yellow pixel, a cyan pixel, and a green pixel, and correction coefficients for each of the yellow, cyan, and green colors may be calculated.

Continuing to refer to FIG. 4 , modeling data representing a relationship between a crosstalk level and a correction coefficient for each color may be generated and stored ( S13 ). Based on the crosstalk level for each target pixel and the tendency of each color correction coefficient calculated in step S12, the relationship between the crosstalk level and each color correction coefficient may be modeled as a linear function or a multi-order function, wherein the function Data representing , for example, coefficients for each order of the function may be generated as modeling data. The modeling data may include data for each of a plurality of target pixels for each of the plurality of grids.

6A, 6B, and 6C are graphs showing the relationship between the crosstalk level and the correction coefficient.

In the graphs of FIGS. 6A to 6C , the horizontal axis indicates the crosstalk level (L _XT ), and the vertical axis indicates a correction coefficient for each color. When the plurality of crosstalk levels and the plurality of color correction coefficients calculated in step S12 of FIG. 4 are divided according to the respective color correction coefficients, the color correction coefficient for the crosstalk level crosstalk level (L _XT ) ( The distribution of Cr, Cg, Cb) may be graphically shown as shown in FIGS. 6A to 6C .

In an embodiment, the crosstalk level L _XT may be a value calculated based on the white image. However, the present invention is not limited thereto, and the crosstalk level L _XT may be a value calculated based on different color images of the multi-light source image obtained in step S11 .

For each color, the _{relationship between the crosstalk level (L XT} ) and the correction coefficient can be modeled as an optimal function through regression analysis. For example, the function F_Cr may be modeled for the red correction coefficient Cr, the function F_Cg may be modeled for the green correction coefficient Cg, and the function F_Cb may be modeled for the blue correction coefficient Cb. The input of the functions F_Cr, F_Cg and F_Cb is the crosstalk level (L _XT ).

In Figures 6a through 6c, but shown as a function of F_Cr, F_Cg and F_Cb mainly the linear function, and thus it is not limited, and each relationship in the cross talk level (L _XT) and the correction factor is to be modeled as a multiple order function can

Continuing to refer to FIG. 4 , for each target pixel of each grid GRD, _{a relationship between the crosstalk level L XT} and each color correction coefficient may be modeled, and each order of the modeled function may be modeled. Coefficients may be generated as modeling data. The modeling data may be stored in a calibration device of an image sensor, for example, a setup device of an image sensor module.

After the modeling data is generated, a single light source calibration step S20 may be performed. The single light source calibration step S20 may be performed for each image sensor module based on the modeling data generated in the multi light source calibration step S10 .

A single light source image may be obtained from the image sensor module (S14). The image sensor module may generate a single light source image by imaging one light source. In an embodiment, the image sensor module may generate a white image by capturing a white light source.

Correction coefficients for each color may be calculated based on the single light source image and modeling data (S15). A crosstalk level of a target pixel may be calculated from a single light source image, for example, a white image, and correction coefficients for each color of the target pixel (eg, Cr, Cg, Cb) can be calculated. For example, based on the modeling data, functions of correction coefficients for each color for the crosstalk level (functions for each of red, green, and blue colors) may be reconstructed, and the crosstalk level of the target pixel is assigned to the restored functions. By applying, correction coefficients for each color of the target pixel may be calculated. For each of the target pixels provided in each of the plurality of grids, color correction coefficients may be calculated based on the corresponding modeling functions and the crosstalk level.

As such, in the single single light source calibration step S20 , correction coefficients for each color may be calculated based on the single light source image obtained by imaging the single light source using the modeling data generated in the multi light source calibration step 10 . The calculated correction coefficients for each color may be stored in a memory in the image sensor module, for example, in the memory 132 of the image sensor 100 of FIG. 1 . During the image sensing operation of the image sensor 100 , the color corrector ( 131 of FIG. 1 ) may perform crosstalk compensation on the target pixel based on color correction coefficients.

As described above, according to the method for calibrating an image sensor according to the present embodiment, in the multi-light source calibration step S10, each of the plurality of image sensor modules generates a plurality of multi-light source images by imaging a plurality of light sources, and Modeling data representing a relationship between a crosstalk level and a correction coefficient for each color may be generated based on the multi-light source image. Thereafter, in order to extract the correction coefficients for each color for a specific image sensor module, a single light source calibration step S20 is performed. may be generated, and correction coefficients for each color of each of the target pixels may be calculated based on the modeling data and the crosstalk level of the target pixels calculated based on the single light source image.

In the calibration step of the image sensor (eg, the calibration step in the mass production step of the image sensor module), when the calibration device extracts a correction coefficient based on the single light source image generated by the image sensor capturing a single light source, various colors are included Appropriate correction may not be made to an image generated by imaging a subject. On the other hand, in order to calculate the correction coefficients for each color, multi-light source images corresponding to the number of colors are required. In the calibration step for the image sensor (or image sensor module), multi-light source images such as red, green and When the blue images are generated and the calibration apparatus calculates the correction coefficients for each color based on them, the burden of installing a plurality of light sources and the burden of multi-light source imaging and multi-light source image processing (eg, processing time, memory, and cost) increase) occurs.

However, according to the calibration method of the image sensor according to the exemplary embodiment of the present disclosure described with reference to FIG. 4 , modeling data representing the relationship between the crosstalk level and the correction coefficient for each color is generated and calibrated in advance based on the multi-light source image. It is stored in the device, and in the calibration step of the image sensor (eg, S20 in FIG. 4 ), the image sensor generates a single light source image by imaging the single light source, and the calibration device calculates the crosstalk level of the target pixel from the single light source image, A correction coefficient for each color of the target pixel may be calculated based on the calculated crosstalk level and modeling data of the target pixel.

Accordingly, additional equipment necessary for multi-light source image acquisition, for example, installation of a plurality of light sources is not required, and the burden on image processing in the calibration step of the image sensor and the time required for calibration (eg, setup time) can be reduced. .

7 illustrates an example of an image sensor calibration apparatus according to an exemplary embodiment of the present disclosure.

Referring to FIG. 7 , the calibration apparatus 2000 may include a calibration processor 2100 and a memory 2200 .

The memory 2200 may store modeling data representing color correction coefficient functions for each of the plurality of target pixels. For example, the modeling data may include coefficients of each order of the color-specific correction coefficient functions. The modeling data generated in the multi-light source calibration step S20 of FIG. 4 may be stored in the memory 2200 . For example, the modeling data may be generated based on a plurality of multi-light source images generated by imaging a plurality of light sources by a plurality of image sensor modules, and the modeling data may include coefficients of each order of the correction coefficient functions for each color. have.

8 shows modeling data according to an exemplary embodiment of the present disclosure.

Referring to FIG. 8 , the modeling data includes the image sensor of the image sensor module 1100 including a pixel array divided into a plurality of grids, and the modeling data includes the grid-specific modeling data (MD_GRD1 to MD_GRDm) for each of the plurality of grids ( m is an integer greater than or equal to 4).

Modeling data for each grid, for example, modeling data MD_GRD1 of the first grid, includes modeling data for each color MDr1, MDg1, MDb1). The color-specific modeling data MDr1 , MDg1 , and MDb1 may include coefficients of respective orders of the color-specific correction coefficient functions for the target pixel. For example, the first color modeling data MDr1 may include coefficients of each order of the correction coefficient function of the first color.

Continuingly, referring to FIG. 7 , the calibration apparatus 2000 receives a single light source image IDT_S from the image sensor module 1100 and the image sensor module 1100 based on the single light source image IDT_S and modeling data. ), correction coefficients Cr, Cg, and Cb for each color for each of the plurality of target pixels may be calculated. In this case, the single light source image IDT_S may be generated by the image sensor module 1100 capturing the single light source 2300 . In an embodiment, the single light source 2300 may be a white light source, and the single light source image IDT_S may be a white image.

The calibration apparatus 2000 calculates the crosstalk level of each of the plurality of target pixels from the single light source image IDT_S, and uses the color correction coefficient functions for each of the plurality of target pixels restored based on the modeling data. By applying the crosstalk level, it is possible to calculate the correction coefficients (Cr, Cg, Cb) for each color.

The calibration apparatus 2000 may store correction coefficients Cr, Cg, and Cb for each color of each of the plurality of target pixels in the image sensor module 1100 . When the image sensor module 1100 performs an image sensing operation, the color correction unit (eg, 131 of FIG. 1 ) of the image sensor crosses the plurality of target pixels based on the respective color correction coefficients (Cr, Cg, Cb) corresponding to each of the plurality of target pixels. Torque correction can be performed.

In an embodiment, the calibration apparatus 2000 may be implemented as a combination of a general-purpose processor such as a micro controller unit (MCU) and a central processing unit (CPU) and firmware (or software module). For example, the firmware or software module may include a program or algorithm including the functions of the above-described calibration apparatus 2000 . However, the present invention is not limited thereto, and the calibration device 2000 may be implemented with hardware logic such as a Field Programmable Gate Array (FPGA), an Application Specific IC (ASIC), a Complex Programmable Logic Device (CPLD), etc. that perform the above-described functions. have.

As described with reference to FIG. 7 , the calibration apparatus 2000 according to the exemplary embodiment of the present disclosure stores modeling data generated in advance based on a plurality of multi-light source images, and captures the image by the image sensor module 1100 . Correction coefficients Cr, Cg, and Cb for each color of the image sensor module 1100 may be calculated based on the obtained single light source image IDT_S and modeling data.

9 shows an example of an apparatus for calibrating an image sensor according to an exemplary embodiment of the present disclosure.

Referring to FIG. 9 , the calibration apparatus 3000 may include a calibration processor 2100 and a memory 2200 . The memory 2200 may store the multi-light source calibration module 3210 and the single light source calibration module 3220 . Here, the multi-light source calibration module 3210 and the single light source calibration module 3220 may be software modules. The multi-light source calibration module 3210 includes a program (or a calibration tool) for performing the multi-light source calibration step (S10 of FIG. 4 ), and the single light source calibration module 3220 performs the single light source calibration step (S20 of FIG. 4 ). It may include a program (or a calibration tool) for performing the .

The processor 3100 may execute the multi-light source calibration module 3210 and the single light source calibration module 3220 to perform the above-described method for calibrating the image sensor according to the exemplary embodiment of the present disclosure. Accordingly, the processor 3100 may calculate correction coefficients (eg, Cr, Cg, and Cb) for each color for crosstalk correction.

The processor 3100 receives a plurality of multi-light source images IDT_M captured from a plurality of arbitrary image sensor modules 1100 by executing the multi-light source calibration module 3210 , and receives the plurality of multi-light source images IDT_M Based on the color-specific correction coefficients and the crosstalk level, the relationship may be modeled as a correction coefficient function, and modeling data, for example, coefficients of each order of the correction coefficient function may be calculated. In an embodiment, the memory 3200 may store modeling data, and the modeling data may be used in a single light source calibration step.

By executing the single light source calibration module 3220, the processor 3100 may control the image sensor module 1100, for example, a specific image sensor module 1100, to generate a single light source image IDT_S by capturing a single light source. . In an embodiment, the processor 3100 receives the single light source image IDT_S from the image sensor module 1100, and color-specific correction coefficients cr, CG, Cb based on the single light source image IDT_S and modeling data ) can be calculated. The calibration apparatus 3000 may store correction coefficients for each color in the image sensor module 1100 .

10 is a block diagram schematically illustrating an electronic system according to an exemplary embodiment of the present disclosure.

Referring to FIG. 10 , the electronic system 3000 may include an image sensor module 3100 and an application processor (AP) 3200 . The electronic system 3000 may be mounted on, for example, a smartphone, and the electronic system 3000 may further include general components, for example, a communication module, a working memory, an input/output module, and a display module.

The image sensor module 3100 may include an image sensor ( 100 of FIG. 1 ), and the image sensor applies color correction coefficients calculated according to the calibration method of the image sensor according to the above-described embodiment of the present disclosure to target pixels. It is stored for each, and during an image sensing operation, crosstalk correction may be performed based on the correction coefficients for each color. Accordingly, the image quality of the image generated by the image sensor may be improved.

The AP 3200 controls the overall operation of the electronic system 3000 and may be implemented as a system-on-chip (SoC) that drives an application program, an operating system, and the like. The AP 3200 may control the image sensor module 3100 , and may generate control signals for controlling the image sensor module 3100 and provide the control signals to the image sensor module 3100 . For example, the AP 3200 may receive a phase difference signal from the image sensor module 3100 and control the image sensor module 3100 to perform auto-focusing based on the received phase difference signal. As described above, the phase difference signal may be generated by the phase detection pixel. The AP 3200 may provide a control signal for auto-focusing to the image sensor module 3100 based on the phase difference. The image sensor module 3100 may perform auto-focusing by moving the lens in a direction in which the distance from the subject increases or decreases based on the control signal.

The AP 3200 may also receive an image from the image sensor module 3100 and perform additional image processing (eg, image quality compensation, processing to increase the dynamic range of the image, gamma compensation, etc.) on the image, and The image-processed image may be stored in a memory or provided to a display module. In addition, the AP 3200 extracts additional information from the image (eg, subject recognition information of the image, the position of the subject in the image) and provides the information to the user or controls the electronic system 3000 based on the information can do.

Exemplary embodiments have been disclosed in the drawings and specification as described above. Although the embodiments have been described using specific terms in the present specification, these are used only for the purpose of explaining the technical spirit of the present disclosure and are not used to limit the meaning or the scope of the present disclosure described in the claims. . Therefore, it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present disclosure should be defined by the technical spirit of the appended claims.

1000: electronic device 100: image sensor
200: processor 110, 110a, 110b: pixel array
120: readout circuit 130: image signal processor
131: color correction unit 132: memory

Claims (10)

In the image sensor calibration method,
acquiring a plurality of multi-light source images generated from a plurality of image sensor modules;
calculating a plurality of crosstalk levels and a plurality of color correction coefficients based on the plurality of multi-light source images;
generating modeling data based on a relationship between a crosstalk level for a first color and the plurality of color-specific correction coefficients; and
Calculating the correction coefficient for each color of the target pixel of the image sensor provided in the first image sensor module based on the single light source image captured by the first image sensor module and the modeling data. The method of claim 1 , wherein calculating the color-specific correction coefficients for the target pixel comprises:
calculating a crosstalk level of the target pixel from the single light source image;
restoring correction coefficient functions for each color corresponding to the target pixel by using the modeling data; and
and calculating the color-specific correction coefficients for the target pixel by applying the crosstalk level of the target pixel to the color-specific correction coefficient functions. 3. The method of claim 2, wherein the first color is white and the single light source image is a white image. The method of claim 1, wherein the correction coefficients for each color are:
and a correction coefficient for each of a plurality of different colors. The method of claim 4, wherein the correction coefficients for each color are:
A method comprising a red correction coefficient, a green correction coefficient and a blue correction coefficient. The method of claim 1 , further comprising: storing the color-specific correction coefficients for the target pixel in the first image sensor module. The method of claim 1, wherein the generating of the modeling data comprises:
modeling a correction coefficient function for each of the correction coefficients for each color through regression analysis on the plurality of crosstalk levels and the correction coefficients for each color; and
and calculating the modeling data representative of the correction factor function. The method of claim 7, wherein the modeling data,
and including a coefficient for each order of the correction coefficient function. According to claim 1,
The target pixel is one of a plurality of pixels disposed around a phase detection pixel provided in the pixel array of the image sensor. According to claim 1, wherein the pixel array of the image sensor,
It includes a plurality of pixels of the same color arranged in a k X k matrix (k is an integer greater than or equal to 2),
wherein the target pixel is at least one of the plurality of pixels.

Images