WO2023016146A1

WO2023016146A1 - Image sensor, image collection apparatus, image processing method, and image processor

Info

Publication number: WO2023016146A1
Application number: PCT/CN2022/103957
Authority: WO
Inventors: 王昂
Original assignee: Oppo广东移动通信有限公司
Priority date: 2021-08-09
Filing date: 2022-07-05
Publication date: 2023-02-16
Also published as: CN113676629A; CN113676629B

Abstract

An image sensor (100), a camera module (200), an image collection apparatus (1000), an image processing method, and an image processor (10). The image sensor (100) comprises a pixel unit array (110) and a filter unit array (120). The pixel unit array (110) comprises a plurality of pixel units (111). The filter unit array (120) comprises a plurality of filter units (121). Each filter unit (121) comprises a plurality of filter sub-units (1211), each filter sub-unit (1211) comprises a first filter (12111) corresponding to one color in a first color space; and at least some filter sub-units (1211) of at least some filter units (121) each further comprise a second filter (12112) corresponding to one color in a second color space. Each first filter (12111) and second filter (12112) of the filter unit (121) cover one photosensitive pixel of a respective pixel unit (111).

Description

Image sensor, image acquisition device, image processing method and image processor

priority information

This application claims the priority and benefit of the patent application No. 202110907046.1 filed with the State Intellectual Property Office of China on August 9, 2021, which is hereby incorporated by reference in its entirety.

technical field

The present application relates to the technical field of image processing, in particular to an image sensor, a camera module, an image acquisition device, an image processing method and an image processor.

Background technique

In recent years, with the rapid popularization of mobile terminals and the continuous development of mobile phone imaging technology, smart phones have increasingly become the most commonly used shooting equipment for the public. The contact image sensor (Contact image sensor, CIS) of the mobile phone uses the color filter array (CFA) covered on its surface to screen the color, and divides the captured light into red, green and blue. A series of calculations are converted into three primary color components and finally synthesized into one color. Among them, white balance, as the first process strongly related to color performance in the entire mobile phone Image Signal Processing (ISP), plays a vital role in the overall tone of the shooting.

At present, the white balance algorithm of the mobile phone mainly relies on two hardwares: one is the original CIS, which roughly estimates the position of the gray area in the picture through the intensity relationship of the RGB channel, and calculates R_gain and B_gain through this position and multiplies them in the whole picture respectively. on the R channel and B channel. The second is to estimate the color temperature value of the entire light source scene through an additional color temperature sensor, and input it to the ISP to make corresponding white balance adjustments. It is difficult for the current color temperature sensor to accurately restore the color temperature through only three channels of red, green and blue, and it is difficult to achieve accurate white balance judgment.

Contents of the invention

In view of this, the present invention aims to solve one of the problems in the related art at least to a certain extent. Therefore, the purpose of the present application is to provide an image sensor, a camera module, an image acquisition device, an image processing method, and an image processor.

The image sensor in the embodiment of the present application includes a pixel unit array and a filter unit array. The pixel unit array includes a plurality of pixel units, each of which includes a plurality of photosensitive pixels. The filter unit array includes a plurality of filter units, each of the filter units includes a plurality of filter subunits, and each of the filter subunits includes a color in the first color space Corresponding to the first filter, at least part of the filter sub-units of at least part of the filter unit also include a second filter corresponding to a color in the second color space, and the filter unit Each of the first optical filter and the second optical filter covers one of the photosensitive pixels corresponding to the pixel unit, and each of the optical filter units includes multiple colors corresponding to the first color space. The first optical filter, the optical filter unit array includes the second optical filter corresponding to a plurality of colors in the second color space.

The camera module in the embodiment of the present application includes a lens and the image sensor.

The image acquisition device in the embodiment of the present application includes an image processor and the image sensor.

The image processing method in the embodiment of the present application includes acquiring light signals collected by an image sensor to process the light signals to generate a first color space image and a second color space image; according to the first color space image and the second color space image The color space image performs white balance processing on the first color space image to obtain a white balance processed image.

The image processor of the present application includes an acquisition module and a white balance processing module. The acquiring module is used to acquire the light signal collected by the image sensor to process the light signal to generate a first color space image and a second color space image; the white balance processing module is used to generate the first color space image according to the first color space image and performing white balance processing on the first color space image with the second color space image to obtain a white balance processed image.

Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

FIG. 1 is a schematic structural diagram of an image sensor in some embodiments of the present application;

Figure 2 is a structural schematic diagram of a four-pixel-in-one array of filter units in an image sensor

3 is a schematic structural diagram of an array of filter units in an image sensor according to some embodiments of the present application;

4 is a schematic structural diagram of an array of filter units in an image sensor according to some embodiments of the present application;

Fig. 5 is a schematic diagram of the matching degree of the human eye response curve corresponding to the image sensor in some embodiments of the present application;

6 is a schematic flow diagram of an image processing method in some embodiments of the present application;

Fig. 7 is a schematic structural diagram of an image acquisition device in some embodiments of the present application;

FIG. 8 is a schematic structural diagram of an image processor in some embodiments of the present application;

FIG. 9 is a schematic flowchart of an image processing method in some embodiments of the present application;

FIG. 10 is a schematic structural diagram of a white balance processing module in an image processor according to some embodiments of the present application;

11 is a schematic diagram of the data logic flow of RGBCMY imaging in some embodiments of the present application;

FIG. 12 is a schematic diagram of space conversion of a CCM matrix in some embodiments of the present application;

Fig. 13 is a schematic flowchart of an image processing method in some embodiments of the present application;

Fig. 14 is a schematic structural diagram of an acquisition module in an image processor in some embodiments of the present application;

FIG. 15 is a schematic structural diagram of an array of filter units in an image sensor according to some embodiments of the present application;

Fig. 16 is a schematic flowchart of an image processing method in some embodiments of the present application;

Fig. 17 is a schematic structural diagram of an image processor in some embodiments of the present application;

Fig. 18 is a schematic structural diagram of a camera module in some embodiments of the present application.

Detailed ways

Embodiments of the present application are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary, are only for explaining the present application, and should not be construed as limiting the present application.

In the description of the present application, it should be understood that the terms "first" and "second" are used for description purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of said features. In the description of the present application, "plurality" means two or more, unless otherwise clearly and specifically defined.

In the description of this application, it should be noted that unless otherwise specified and limited, the terms "installation", "connection", and "connection" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection. Connected, or integrally connected; may be mechanically connected, may be electrically connected or may communicate with each other; may be directly connected, or indirectly connected through an intermediary, may be internal communication between two components or interaction between two components relation. Those of ordinary skill in the art can understand the specific meanings of the above terms in this application according to specific situations.

The following disclosure provides many different implementations or examples for implementing different structures of the present application. To simplify the disclosure of the present application, components and arrangements of specific examples are described below. Of course, they are examples only and are not intended to limit the application. Furthermore, the present application may repeat reference numerals and/or reference letters in various instances, such repetition is for simplicity and clarity and does not in itself indicate a relationship between the various embodiments and/or arrangements discussed.

Embodiments of the present application provide an image sensor. The image sensor includes a pixel unit array and a filter unit array. The pixel unit array includes a plurality of pixel units, each of which includes a plurality of photosensitive pixels. The filter unit array includes a plurality of filter units, each of the filter units includes a plurality of filter subunits, and each of the filter subunits includes a color in the first color space Corresponding to the first filter, at least part of the filter sub-units of at least part of the filter unit also include a second filter corresponding to a color in the second color space, and the filter unit Each of the first optical filter and the second optical filter covers one of the photosensitive pixels corresponding to the pixel unit, and each of the optical filter units includes multiple colors corresponding to the first color space. The first optical filter, the optical filter unit array includes the second optical filter corresponding to a plurality of colors in the second color space.

In some embodiments, a plurality of the filter units are arranged in a rectangular array, and a plurality of the filter subunits in the filter unit are arranged in a rectangular array.

In some embodiments, each of the filter subunits includes at least one first filter corresponding to one color and at least one second filter corresponding to one color.

In some embodiments, each of the filter subunits covers 2*2 of the photosensitive pixels corresponding to the pixel unit, and the first filter along the first filter subunit Diagonal distribution, the second optical filter is distributed along the second diagonal of the optical filter subunit, or the first optical filter and the second optical filter are arranged side by side along the same direction.

In some embodiments, the first color space includes red, green, and blue, and the second color space includes cyan, magenta, and yellow.

In some embodiments, each filter unit includes 2*2 filter subunits, one filter subunit with a red filter and one filter subunit with a blue filter The filter subunits are distributed along a first diagonal of the filter unit, and two filter subunits with green filters are distributed along a second diagonal of the filter unit.

In some embodiments, the first color space includes red, green, and blue, the second color space includes cyan, magenta, and yellow, and one of the filter subunits includes a green filter and a yellow filter. A filter, or one of the filter subunits includes a red filter and a magenta filter, or one of the filter subunits includes a blue filter and a cyan filter.

In some embodiments, the first color space includes red, green and blue. When the filter unit only includes the first filter corresponding to the red, the green and the blue, the filter unit is arranged in a four-in-one Bayer array; or, When the filter unit includes a plurality of first filters and one or more second filters, the first filter subunit and the second filter subunit form a first diagonal distribution, and the two third filter subunits are distributed in a second diagonal. Wherein, the first filter subunit only includes a red filter, or the first filter subunit includes the red filter and the color corresponding to one color in the second color space. second filter; the second filter subunit includes only a blue filter, or the second filter subunit includes a blue filter and a color in the second color space The corresponding second filter; the third filter subunit only includes a green filter, and/or, the third filter subunit includes a green filter and the second color space A color corresponds to the second filter.

An embodiment of the present application provides a camera module. A camera module includes a lens and an image sensor. The image sensor includes a pixel unit array and a filter unit array. The pixel unit array includes a plurality of pixel units, each of which includes a plurality of photosensitive pixels. The filter unit array includes a plurality of filter units, each of the filter units includes a plurality of filter subunits, and each of the filter subunits includes a color in the first color space Corresponding to the first filter, at least part of the filter sub-units of at least part of the filter unit also include a second filter corresponding to a color in the second color space, and the filter unit Each of the first optical filter and the second optical filter covers one of the photosensitive pixels corresponding to the pixel unit, and each of the optical filter units includes multiple colors corresponding to the first color space. The first optical filter, the optical filter unit array includes the second optical filter corresponding to a plurality of colors in the second color space.

An embodiment of the present application provides an image acquisition device. The image acquisition device includes an image processor and the image sensor described in any one of the above-mentioned implementation manners.

An embodiment of the present application provides an image processing method. The image processing method includes: acquiring a light signal collected by an image sensor to process the light signal to generate a first color space image and a second color space image; according to the first color space image and the second color space image performing white balance processing on the image in the first color space to obtain a white balance processed image.

In some embodiments, the acquiring the light signal collected by the image sensor to process the light signal to generate the first color space image and the second color space image includes: determining according to the pixel arrangement of the image sensor Correlation between the first color space image and the second color space image; processing the light signal according to the correlation and interpolation algorithm to obtain the first color space image and the second color space image .

In some embodiments, performing white balance processing on the first color space image according to the first color space image and the second color space image to obtain a white balance processed image includes: performing the white balance processing on the first color space image Preprocessing the color space image to obtain a first preprocessing image, and processing the second color space image through a virtual channel to obtain a second preprocessing image; summarizing the second preprocessing image to the first preprocessing The image is obtained by performing white balance processing on the first pre-processing image to obtain a white balance processed image.

In some embodiments, the preprocessing includes black level compensation processing, lens correction processing and/or dead point compensation processing.

In some embodiments, the image processing method further includes: performing color interpolation processing and color correction processing on the white balance processed image to obtain an output image.

In some implementations, the image processor includes: an acquisition module, configured to acquire light signals collected by the image sensor to process the light signals to generate a first color space image and a second color space image a white balance processing module, configured to perform white balance processing on the first color space image according to the first color space image and the second color space image to obtain a white balance processing image.

It is difficult for the current color temperature sensor to accurately restore the color temperature through only three channels of red, green and blue, and it is difficult to achieve accurate white balance judgment.

In view of this, embodiments of the present application provide an image sensor, an image acquisition device, and an image processing method.

Referring to FIG. 1 , an embodiment of the present application provides an image sensor 100 . The image sensor 100 includes: a pixel unit array 110 and a filter unit array 120 .

The pixel unit array 110 includes a plurality of pixel units 111 , and each pixel unit 111 includes a plurality of photosensitive pixels 1111 . The filter unit array 120 includes a plurality of filter units 121 . Each filter unit 121 includes a plurality of filter subunits 1211, and each filter subunit 1211 includes a first filter 12111 corresponding to a color in a first color space, at least some of the filters At least part of the filter subunit 1211 of the unit 121 further includes a second filter 12112 corresponding to a color in a second color space. Each of the first filter 12111 and the second filter 12112 of the filter unit 121 covers a photosensitive pixel of the corresponding pixel unit 111. Each filter unit 121 includes first filters 12111 corresponding to multiple colors in the first color space, and the filter unit array 120 includes second filters 12112 corresponding to multiple colors in the second color space.

Specifically, the first color space includes red (R), green (G) and blue (B), and the second color space includes cyan (C), magenta (M) and yellow (Y). The first color space is the RGB color space, and the second color space is the CMY color space as an example for illustration. Of course, the first color space and the second color space may also be other color spaces, which will not be described here.

At least some of the filter subunits 1211 of at least some of the filter units 121 further include a second filter 12112 corresponding to one color in a second color space, which can be understood as including two cases. The first case may be that all the filter subunits 1211 of all the filter units 121 as shown in FIG. 3 include the second filter 12112 . The second case is that only part of the filter subunit 1211 of part of the filter unit 121 as shown in FIG. The optical sheet sub-unit 1211 may also not include the second optical filter 12112 . That is, each filter subunit 1211 includes a first filter 12111 corresponding to a first color space, and at least part of the filter subunits 1211 of at least part of the filter unit 121 also includes a second color space The corresponding second optical filter 12112 refers to that part of the optical filter subunit 1211 in the image sensor 100 of the present application may include the first optical filter 12111 and the second optical filter 12112 (as shown in FIG. 3 ), some of the filter subunits 1211 may only include the first filter 12111 (as shown in FIG. 12 ).

It can be understood that the more the number of the first optical filter 12111 and the second optical filter 12112 are at the same time, the more the number of spectral channels is correspondingly increased, which can increase the white balance reference information in the subsequent white balance processing. Therefore, the accuracy of gray point detection and light source analysis can be greatly improved, and a more accurate white balance judgment of the image sensor 100 can be realized.

In addition, the image sensor of the present application has six kinds of RGBCMY filter materials, which is the most mature so far, and also the largest number of color channels that can be achieved by this technical route. This absorbing filter brings stable filtering characteristics, which can effectively solve the problem of unstable filtering characteristics caused by changes in the filtering characteristics with the incident angle of light and ambient temperature.

Referring to FIG. 1 , each pixel unit 111 of the image sensor 100 includes 2*2 photosensitive pixels 1111 . Each filter subunit 1211 covers 2*2 photosensitive pixels 1111 in the corresponding pixel unit 111 , and each filter ( 12111 or 12112 ) of the filter subunit 1211 covers one photosensitive pixel 1111 . Wherein, each filter subunit 1211 includes a first filter 12111 corresponding to a first color space, and at least part of the filter subunits 1211 also includes a second filter 12112 corresponding to a second color space , each filter of the filter unit 121 covers one photosensitive pixel of the corresponding pixel unit 111 . A plurality of filter units 121 constitute a filter unit array 120 .

In the image sensor 100 of the present application, in some of the filter subunits 1211 in the filter array 1200, the number of spectral channels can be increased by replacing all the first filters 12111 with some of the second filters 12112 , so that the reference information of the white balance is increased, thereby greatly improving the accuracy of gray point detection and light source analysis, so as to realize a more accurate white balance judgment of the image sensor 100 .

In the filter unit array 120, some structures have a plurality of filter units 121 corresponding to filters in different first color spaces in a four-pixel-in-one Bayer array (quad-bayer) arrangement (as shown in FIG. 2 ), quad -bayer can combine squares of four pixels into one color. In this application, based on the quad-bayer array arrangement, half of each group of 2x2 R, G, and B (diagonal directions) can be replaced by M, Y, and C (as shown in FIG. 3 ).

Referring to FIG. 2 or FIG. 3 , the plurality of filter units 121 of the present application are arranged in a rectangular array, and the plurality of filter subunits 1211 in the filter unit 121 are arranged in a rectangular array.

In addition, the arrangement of 2*2 photosensitive pixels 1111 in the image sensor 100 can adapt to some new technologies that have been widely used in mobile phone image sensors, including binning mode, 2x2 large microlens (2x2OCL) , full-pixel dual-core focus (dual PD) and QDOL technologies, while improving the white balance judgment accuracy of the image sensor 100, the adaptability of the image sensor 100 to new technologies can be improved.

The image sensor 100 of the present application adds a second color space on the basis of the first space color space to double the number of spectral channels. Doubling the number of spectral channels means doubling the white balance reference information, which can greatly improve gray point detection. And the accuracy of light source analysis, to achieve more accurate white balance judgment.

Referring to FIG. 1 or FIG. 3, in some embodiments, each filter subunit 1211 includes at least one first filter 12111 corresponding to one color and at least one second filter 12111 corresponding to one color. Tablet 12112.

That is to say, the distribution of the RGB colors and CMY colors inside the filter subunit 1211 may include various distribution methods, such as regular (diagonal distribution as shown in FIG. 3 ), or other different distribution methods. The regular distribution method only needs to satisfy that each filter subunit 1211 has at least one first filter 12111 corresponding to one color and at least one second filter 12112 corresponding to one color. No limit.

In addition, the number of second filters 12112 in each filter subunit 1211 can also be more than that of the first filters 12111, for example, in the case of an arrangement of 3*3 photosensitive pixels, the second The number of the first optical filter 12111 and the second optical filter 12112 may not be equal, and the corresponding distribution of the first optical filter 12111 and the second optical filter 12112 may also have various situations.

Please refer to FIG. 1 and FIG. 3 together. In some embodiments, the first optical filter 12111 is distributed along the first diagonal of the optical filter subunit 1211, and the second optical filter 12112 is distributed along the first diagonal of the optical filter subunit 1211. The second diagonal distribution of . Or, referring to FIG. 4 , the first filter 12111 and the second filter 12112 are arranged side by side along the same direction. The above arrangements can realize uniform distribution of spectral channels corresponding to the first color space and spectral channels corresponding to the second color space in the filter unit in the image sensor, which is beneficial to the regular arrangement of the filter array 120 . In other embodiments of the present application, the arrangement of the filters in the filter array 120 may also be arranged irregularly, which is not limited here.

More specifically, please refer to FIG. 3 again. In some embodiments of the present application, each filter unit 121 includes 2*2 filter subunits 1211, one with a red filter (R) The optical filter subunit 1211 and a filter subunit 1211 with a blue filter (B) are distributed along the first diagonal of the filter unit 1211, and two filter subunits with a green filter (G) The subunits 1211 are distributed along the second diagonal of the filter unit 1211 .

It should be noted that the first diagonal distribution and the second diagonal distribution are only used to illustrate that the diagonal distribution directions of the first optical filter 12111 and the second optical filter 12112 are inconsistent, and do not refer to a fixed diagonal direction.

Compared with the original spectral channels corresponding to the first color space, this application replaces half of the filter units with the second color space and increases the spectral channels corresponding to the second color space, which can double the number of spectral channels Doubling the number of spectral channels means doubling the white balance reference information, which will greatly improve the accuracy of gray point detection and light source analysis, and achieve more accurate white balance judgments.

More specifically, referring to FIG. 3, a filter subunit 1211 may only include a green filter (G) and a yellow filter (Y), or a filter subunit 1211 may also include only a red filter. The light filter (R) and the magenta color filter (M), or one filter subunit 1211 may also include only the blue color filter (B) and the cyan color filter (C).

At this time, the image sensor 100 has 6 color channels (RGBCMY), which can greatly improve the matching degree of fitting the human eye response curve (as shown in FIG. 5 ), which also means that the color reproduction The performance is more accurate and closer to the actual perception of the human eye.

Please refer to FIG. 15 , in some embodiments of the present application, the first color space includes red (R), green (G) and blue (B). When the filter unit 121 only includes first filters 12111 corresponding to red (R), green (G) and blue (B), the filter unit 121 is arranged in a four-in-one Bayer array. When the filter unit 121 includes a plurality of first filters 12111 and one or more second filters 12112, the first filter subunit 1211a and the second filter subunit 1211b form a first diagonal distribution, the two third filter subunits 1211c are distributed in a second diagonal.

Wherein, the first filter subunit 1211a may only include a red filter (R), or the first filter subunit 1211a may include a red filter (R) and a first color corresponding to a color in the second color space Two filters 12112, the second filter subunit 1211b may only include a blue filter (B), or the second filter subunit 1211b may also include a blue filter (B) and a second color A second filter 12112 corresponding to one color in space. The third filter sub-unit 1211c only includes a green filter (G), and/or, the third filter sub-unit 1211c includes a green filter (G) and a first color corresponding to a color in the second color space. Two filters 12112. Understandably, this description may include the following situations:

First, the first filter subunit 1211a only includes a red filter (R), the second filter subunit 1211b may only include a blue filter (B), and the third filter subunit 1211c Only the case of the green filter (G) is included. At this time, the filter units 121 are distributed in a four-in-one Bayer array.

In the second type, the first filter subunit 1211a only includes a red filter (R), and the second filter subunit 1211b includes a blue filter (B) corresponding to a color in the second color space. The second filter 12112 and the third filter subunit 1211c only include a green filter (G) (as shown in FIG. 15 ).

Third, the first filter subunit 1211a includes a red filter (R) and a second filter 12112 corresponding to one color in the second color space, and the second filter subunit 1211b only includes blue Filter (B), the third filter sub-unit 1211c includes only the green filter (G).

Fourth, the first filter subunit 1211a includes a red filter (R) and a second filter 12112 corresponding to a color in the second color space, and the second filter subunit 1211b includes a blue filter The light sheet (B) corresponds to a second color filter 12112 in the second color space, and the third filter subunit 1211c only includes a green filter (G).

Fifth, the first filter subunit 1211a only includes a red filter (R), the second filter subunit 1211b may only include a blue filter (B), and the third filter subunit 1211c It includes a green filter (G) and a second filter 12112 corresponding to one color in the second color space.

Sixth, the first filter subunit 1211a only includes a red filter (R), and the second filter subunit 1211b includes a blue filter (B) corresponding to one color in the second color space The second filter 12112 and the third filter subunit 1211c include a green filter (G) and a second filter 12112 corresponding to one color in the second color space.

Seventh, the first filter subunit 1211a includes a red filter (R) and a second filter 12112 corresponding to one color in the second color space, and the second filter subunit 1211b only includes blue The filter (B), the third filter subunit 1211c includes a green filter (G) and a second filter 12112 corresponding to one color in the second color space.

Eighth, the first filter subunit 1211a includes a red filter (R) and a second filter 12112 corresponding to one color in the second color space, and the second filter subunit 1211b includes a blue filter The light sheet (B) corresponds to a second color filter 12112 in the second color space, and the third filter subunit 1211c includes a green filter (G) corresponding to a color in the second color space. Second filter 12112.

Understandably, the second color space may include cyan (C), magenta (M) and yellow (Y). Different color arrangements can obtain different spectral channel information, thereby obtaining different spectral communication information, enhancing the diversity of white balance reference information, and providing a variety of ways to improve the accuracy of gray point detection and light source analysis. More options to achieve more accurate white balance judgment.

It should be noted that, FIG. 15 only shows the situation that the filter unit 121 includes a plurality of first filters 12111 and one second filter 12112 . The positions of the first filter sub-unit 1211a and the second filter sub-unit 1211b can be interchanged, which is not limited here.

Please refer to FIG. 6 , the present application also provides an image processing method. Image processing methods include:

01: Obtain the light signal collected by the image sensor to process the light signal to generate a first color space image and a second color space image;

03: Perform white balance processing on the first color space image according to the first color space image and the second color space image to obtain a white balance processed image.

Referring to FIG. 7 , the present application also provides an image acquisition device 1000 . The image acquisition device 1000 includes an image processor 10 and an image sensor 100 , and the image processor 10 and the image sensor 100 can be electrically connected. The image sensor 100 in the image acquisition device 100 of the present application adds a second color space to double the number of spectral channels. Doubling the number of spectral channels means doubling the white balance reference information, which will greatly improve the accuracy of gray point detection and light source analysis. , to achieve more accurate white balance judgment. In some embodiments, the image capture device 100 may be a camera, a mobile phone, a computer, and other devices that capture images.

Please refer to FIG. 8 , the image processor 10 includes an acquisition module 11 and a white balance processing module 13 .

Step 01 can be realized by the acquisition module 11 , and step 03 can be realized by the white balance processing module 13 . That is, the obtaining module 11 is used to obtain the light signal collected by the image sensor to process the light signal to generate the first color space image and the second color space image; the white balance processing module 13 is used to obtain the light signal according to the first color space image and the second color space image The second color space image performs white balance processing on the first color space image to obtain a white balance processed image.

Specifically, since the image sensor 100 has a first filter 12111 corresponding to the first color space and a second filter 12112 corresponding to the second color space, after the acquisition module 11 acquires the collected light, the light signal passes through the second A filter 12111 can generate a first color space image, and a light signal passing through a second filter 12112 can generate a second color space image.

Wherein, referring to Fig. 9, step 03 includes:

031: Preprocessing the first color space image to obtain a first preprocessing image, and processing the second color space image through a virtual channel to obtain a second preprocessing image;

032: Summarize the second pre-processed image to the first pre-processed image to perform white balance processing on the first pre-processed image to obtain a white balance processed image.

Please refer to FIG. 10 , the white balance processing module 13 includes a preprocessing unit 131 and a white balance processing unit 132 .

Step 031 can be implemented by the preprocessing unit 131 , and step 132 can be implemented by the white balance processing unit 132 . That is to say, the preprocessing unit 131 is used to preprocess the first color space image to obtain a first preprocessed image, and process the second color space image through a virtual channel to obtain a second preprocessed image; the white balance processing unit 132 is configured to aggregate the second pre-processed image into the first pre-processed image to perform white balance processing on the first pre-processed image to obtain a white balance processed image.

That is to say, the preprocessing unit 131 can preprocess the first color space image to obtain the first preprocessed image, wherein the preprocessing method includes black level compensation processing, lens correction processing and/or dead point compensation processing, that is, preprocessing The treatment methods can be one of them, two or more than two. In other embodiments of the present application, the preprocessing methods may also include processing processes such as denoising, high dynamic range, and lens correction, and required preprocessing methods may be added according to user requirements, which will not be described here.

The main purpose of image preprocessing is to eliminate irrelevant information in the image, restore useful real information, enhance the detectability of relevant information, and simplify data to the greatest extent, thereby improving the reliability of feature extraction, image segmentation, matching and recognition.

The preprocessing process can include three processes of grayscale, geometric transformation and image enhancement.

Specifically, grayscale: For example, in the RGB model, if R=G=B, the color represents a grayscale color, where the value of R=G=B is called the grayscale value, therefore, each grayscale image A pixel only needs one byte to store the grayscale value (also known as intensity value, brightness value), and the grayscale range is 0-255. Generally, there are four methods: component method, maximum value method, average value method, weighted average method, and four methods to grayscale the color image. When processing color images, we often need to process the three channels sequentially.

Geometric transformation: image geometric transformation, also known as image space transformation, processes the collected image through geometric transformations such as translation, transposition, mirroring, rotation, and zooming, and is used to correct the system error of the image acquisition system and the position of the instrument (imaging angle, Perspective relationship and even the random error of the lens itself). In addition, a grayscale interpolation algorithm needs to be used, because the pixels of the output image may be mapped to the non-integer coordinates of the input image for calculation according to this transformation relationship. Commonly used methods are nearest neighbor interpolation, bilinear interpolation and bicubic interpolation.

Image enhancement: enhancing the useful information in the image, it can be a process of distortion, its purpose is to improve the visual effect of the image, for the application of a given image, purposefully emphasize the overall or local characteristics of the image, and transform the original A clear image becomes clear or emphasizes some interesting features, expands the difference between different object features in the image, suppresses uninteresting features, improves image quality, enriches information, strengthens image interpretation and recognition effects, and satisfies Some special analysis needs. Image enhancement algorithms can be divided into two categories: spatial domain methods and frequency domain methods. The spatial domain method is a direct image enhancement algorithm, which is divided into a point operation algorithm and a neighborhood denoising algorithm. The point operation algorithm is gray level correction, gray level transformation (also known as contrast stretching) and histogram correction. Neighborhood enhancement algorithms are divided into image smoothing and sharpening. Common smoothing algorithms include mean filtering, median filtering, and spatial filtering. Common sharpening algorithms include gradient operator method, second derivative operator method, high-pass filter, mask matching method, etc. The frequency domain method is an indirect image enhancement algorithm, and the commonly used frequency domain enhancement methods include low-pass filter and high-pass filter. Low-frequency filters include ideal low-pass filters, Butterworth low-pass filters, Gaussian low-pass filters, and exponential filters. High-pass filters include ideal high-pass filters, Butterworth high-pass filters, Gaussian high-pass filters, and exponential filters.

The preprocessing unit 131 can perform the above preprocessing on the first color space image (RGB color space image) through the RGB channel, that is, perform the conventional processing of the image signal processing (ISP) process on the first color space image, and the processed first color space image can be obtained A preprocessed image (RGB preprocessed image).

In addition, the preprocessing unit 131 may also process the second color space image (CMY color space image) through virtual channels to obtain a second preprocessing image (CMY preprocessing image). Wherein, the virtual channel may be as shown in FIG. 11 .

The preprocessing method of the second color space image is consistent with the preprocessing method of the first color space image, and can also include black level compensation, lens correction, dead point correction, one, two or more of the processing processes, which can be It is ensured that the processing effects of the two pre-processed images are the same, so that when the aggregated pre-processed images are subjected to white balance processing, the white balance reference information obtained by the two color space images can be directly aggregated.

Specifically, the virtual channel (Virtual channel) can make the CMY color space image and the RGB color space image be processed in parallel or time-sharing at the same time, and are aggregated to the RGB channel at the white balance processing node, that is, the white balance processing module 13 performs the second preprocessing The images are aggregated into the first pre-processing image to perform white balance processing on the first pre-processing image to obtain a white balance processing image.

It can be understood that processing the CMY color space image and the RGB color space image in parallel at the same time can speed up the image signal processing process and improve the efficiency of the image signal processing process. Time-sharing processing of CMY color space image and RGB color space image can make the processing process of CMY color space image and the processing process of RGB color space image interleaved, it is not easy to confuse the processing information of color space image, and ensure that the two color space image information ( RGB information and CMY information) processing accuracy.

The functions of the two pre-processed images after the summary pre-processing in the white balance processing node include:

a. CMY information and RGB information are summarized to form 6-channel information, which can greatly improve the accuracy of light environment judgment. The methods of environment judgment include gray point detection and light source analysis, which can improve the accuracy of data such as gray point detection or light source analysis. Accuracy;

b.6 channel information can participate in the color restoration (as shown in Figure 11) process of the white balance process, and the original color correction matrix (Color Correction Matrix, CCM) used for the color correction (as shown in Figure 11) process Expanding from 3x3 to 6x3 can better fit the response curve of the human eye (as shown in Figure 5), thereby greatly improving the accuracy of color reproduction in white balance processing.

Wherein, CCM is a matrix composed of at least two matrices. As shown in FIG. 8 , the general image signal processing (ISP) flow is subjected to CCM processing after demosaicing (Demosaic). Taking RGB information as an example, CCM can complete the conversion from sensor_RGB space to XYZ space, and then to linear sRGB space. After CCM processing, Gamma correction is performed on the image. After Gamma, the space in the image signal processing becomes a nonlinear sRGB space.

Please refer to the space conversion schematic diagram in Figure 12, the CCM matrix M algorithm process is as follows: the sensor_RGB space is completed by matrix M ² and matrix M ¹ and γ correction. Among them, the sensorRGB space is called the "source color space", and the nonlinear sRGB space is called the "target color space". At present, we can get the 24-color color blocks corresponding to the "unsaturated image" of the source color space (sensorRGB space) , there is also a 24-color color block corresponding to the "saturation image" in the nonlinear sRGB space, and the values of the matrix M ¹ and γ are known, then it is only necessary to inversely correct the image in the nonlinear sRGB space and then convert it to XYZ space, then the matrix M ² can be obtained by combining with the sensorRGB value, and then the color correction matrix M (M=M ² *M ¹ ) can be obtained.

Referring to Figure 13, in some embodiments, step 01 includes:

013: Determine the correlation between the first color space image and the second color space image according to the pixel arrangement of the image sensor;

015: Process the light signal according to the correlation and interpolation algorithm to obtain the first color space image and the second color space image.

Please refer to FIG. 14 , the acquisition module 11 includes a determination unit 113 and a processing unit 115 .

Step 013 can be implemented by the determining unit 113 , and step 015 can be implemented by the processing unit 115 . That is, the determination unit 111 is used to determine the correlation between the first color space image and the second color space image according to the pixel arrangement of the image sensor; the processing unit 112 is used to process the light signal according to the correlation and interpolation algorithm to obtain the first color space image. A color space image and a second color space image.

It can be understood that cyan (C), magenta (M) and yellow (Y) exist as complementary colors of red, green and blue, and can also be used as the three primary colors of pigments, which is called the CMY chromaticity model. There is a correlation between C, M, Y and R, G, B, and the first color space image (RGB color space image) and the second color space image (CMY color space image) can be determined according to the pixel arrangement of the image sensor 100 correlation between. For example, the arrangement of the pixels of the pixel unit array 110 of the image sensor 100 may be the arrangement shown in FIG. 3 , or the arrangement shown in FIG. The arrangement in Fig. 15 can be obtained), or other arrangements. When the pixel arrangement is as shown in FIG. 3 , the red photosensitive pixels in the third row and first column in FIG. 3 are surrounded by yellow photosensitive pixels, cyan photosensitive pixels and cyan photosensitive pixels respectively. The magenta photosensitive pixels in the first row and the third column in FIG. 3 are surrounded by yellow photosensitive pixels, blue photosensitive pixels and blue photosensitive pixels respectively. Therefore, the color space image corresponding to the magenta photosensitive pixel in the first row and the third column in Fig. An image in the second color space relative to the first color space.

Then, the light signal is processed according to the correlation method and the interpolation algorithm to obtain the first color space image and the second color space image. There are many interpolation algorithms for image processing, including nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc. Among them, after weighing speed and quality, the best is the linear interpolation compensation algorithm.

This application uses the correlation processing between RGB and CMY to obtain the first color space image and the second color space image. Compared with the traditional RGB image processing mode, it can ensure that no more human influence is introduced and the accuracy of the processed image is guaranteed.

Referring to Figure 16, in some embodiments, the image processing method further includes:

05: Perform color interpolation processing and color correction processing on the white balance processed image to obtain an output image.

Please refer to FIG. 17 , the image processor 10 further includes a color processing module 15 .

The color processing module 15 is used for performing color interpolation processing and color correction processing on the white balance processed image to obtain an output image.

Specifically, take the Bayer CFA type color filter array (as shown in Figure 2) as an example. In this image array, each pixel has only one color hue value, and the other two color hues must use the The correlation of is obtained by calculation, and this calculation method is called color interpolation algorithm.

The color interpolation process can use a bilinear interpolation algorithm to realize the conversion between the RGB color image and the Bayer-type color filter array map, and the conversion between the CMY color image and the corresponding color filter array map.

Color correction processing, commonly known as color calibration (Color calibration), it can ensure that the color of the image can be more accurately reproduced what the human eye sees at the shooting scene, making the white balance processed image closer to the actual perception of the human eye.

Please refer to FIG. 18 , the present application also provides a camera module 200 . The camera module 200 includes a lens 210 and an image sensor 100 . The image sensor 100 in the camera module 200 of the present application adds a second color space to double the number of spectral channels, and doubling the number of spectral channels means doubling the white balance reference information, which will greatly improve the accuracy of gray point detection and light source analysis To achieve more accurate white balance judgment.

The above examples only express several implementation modes of the present application, and the description thereof is relatively specific and detailed, but should not be construed as limiting the patent scope of the present application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the scope of protection of the patent application should be based on the appended claims.

Claims

An image sensor, including:

a pixel unit array, the pixel unit array including a plurality of pixel units, each of which includes a plurality of photosensitive pixels; and

An array of filter units, the array of filter units includes a plurality of filter units, each of the filter units includes a plurality of filter subunits, and each of the filter subunits includes a first A first filter corresponding to one color in the color space, at least part of the filter subunits of at least part of the filter unit further includes a second filter corresponding to one color in the second color space, Each of the first filter and the second filter of the filter unit covers one of the photosensitive pixels corresponding to the pixel unit, and each filter unit includes a first color The first optical filters correspond to multiple colors in the space, and the filter unit array includes the second optical filters corresponding to multiple colors in the second color space.
The image sensor according to claim 1, wherein a plurality of the filter units are arranged in a rectangular array, and a plurality of the filter subunits in the filter unit are arranged in a rectangular array.
The image sensor according to claim 1, wherein each of the filter subunits includes at least one first filter corresponding to one color and at least one second filter corresponding to one color. light sheet.
The image sensor according to claim 3, wherein each of the filter sub-units covers 2*2 of the photosensitive pixels corresponding to the pixel unit, and the first filter along the filter The first diagonal distribution of the sheet sub-unit, the second optical filter is distributed along the second diagonal of the optical filter sub-unit, or the first optical filter and the second optical filter are distributed along the same Directions are set side by side.
The image sensor according to any one of claims 1-4, wherein the first color space includes red, green and blue, and the second color space includes cyan, magenta and yellow.
The image sensor according to claim 5, wherein each filter unit comprises 2*2 said filter subunits, one with a red filter and one with a blue filter. The filter subunits of the filter are distributed along the first diagonal of the filter unit, and the two filter subunits with green filters are distributed along the second diagonal of the filter unit. Diagonal distribution.
The image sensor according to claim 3, wherein said first color space includes red, green, and blue, said second color space includes cyan, magenta, and yellow, and one of said filter subunits includes green filter and a yellow filter, or one of said filter subunits includes a red filter and a magenta filter, or one of said filter subunits includes a blue filter and a cyan filter .
The image sensor of claim 1, wherein the first color space includes red, green and blue,

When the filter unit only includes the first filter corresponding to the red, the green and the blue, the filter unit is arranged in a four-in-one Bayer array; or

When the filter unit includes a plurality of first filters and one or more second filters, the first filter subunit and the second filter subunit form a first pair Angular distribution, the two third filter subunits are distributed in the second diagonal;

Wherein, the first filter subunit only includes a red filter, or the first filter subunit includes the red filter and the color corresponding to one color in the second color space. second filter;

The second filter subunit only includes a blue filter, or the second filter subunit includes a blue filter and the second color corresponding to one color in the second color space. filter;

The third filter subunit only includes a green filter, and/or, the third filter subunit includes a green filter and the second filter corresponding to one color in the second color space light sheet.
A camera module, including a lens and an image sensor, and the image sensor includes:

a pixel unit array, the pixel unit array including a plurality of pixel units, each of which includes a plurality of photosensitive pixels; and

An array of filter units, the array of filter units includes a plurality of filter units, each of the filter units includes a plurality of filter subunits, and each of the filter subunits includes a first A first filter corresponding to one color in the color space, at least part of the filter subunits of at least part of the filter unit further includes a second filter corresponding to one color in the second color space, Each of the first filter and the second filter of the filter unit covers one of the photosensitive pixels corresponding to the pixel unit, and each filter unit includes a first color The first optical filters correspond to multiple colors in the space, and the filter unit array includes the second optical filters corresponding to multiple colors in the second color space.
The camera module according to claim 9, wherein a plurality of the filter units are arranged in a rectangular array, and a plurality of the filter subunits in the filter unit are arranged in a rectangular array.
The camera module according to claim 9, wherein each of the filter subunits includes at least one first filter corresponding to one color and at least one second filter corresponding to one color. filter.
The camera module according to claim 11, wherein each of the filter subunits covers 2*2 of the photosensitive pixels corresponding to the pixel unit, and the first filter is along the filter The first diagonal distribution of the sheet sub-unit, the second optical filter is distributed along the second diagonal of the optical filter sub-unit, or the first optical filter and the second optical filter are distributed along the same Directions are set side by side.
The camera module according to any one of claims 9-11, wherein the first color space includes red, green and blue, and the second color space includes cyan, magenta and yellow.
The camera module according to claim 13, wherein each filter unit includes 2*2 said filter subunits, one with a red filter and one with a blue filter. The filter subunits of the color filter are distributed along the first diagonal of the filter unit, and the two filter subunits with green filters are distributed along the second diagonal of the filter unit Diagonal distribution.
The camera module according to claim 11, wherein the first color space includes red, green, and blue, the second color space includes cyan, magenta, and yellow, and one filter subunit includes A green filter and a yellow filter, or one of the filter subunits includes a red filter and a magenta filter, or one of the filter subunits includes a blue filter and a cyan filter piece.
The camera module according to claim 9, wherein the first color space includes red, green and blue,

When the filter unit only includes the first filter corresponding to the red, the green and the blue, the filter unit is arranged in a four-in-one Bayer array; or

When the filter unit includes a plurality of first filters and one or more second filters, the first filter subunit and the second filter subunit form a first pair Angular distribution, the two third filter subunits are distributed in the second diagonal;

Wherein, the first filter subunit only includes a red filter, or the first filter subunit includes the red filter and the color corresponding to one color in the second color space. second filter;

The second filter subunit only includes a blue filter, or the second filter subunit includes a blue filter and the second color corresponding to one color in the second color space. filter;

The third filter subunit only includes a green filter, and/or, the third filter subunit includes a green filter and the second filter corresponding to one color in the second color space light sheet.
An image acquisition device, comprising an image processor and the image sensor according to any one of claims 1-8.
An image processing method, wherein the image processing method includes:

Acquiring light signals collected by the image sensor to process the light signals to generate a first color space image and a second color space image;

Performing white balance processing on the first color space image according to the first color space image and the second color space image to obtain a white balance processed image.
The image processing method according to claim 18, wherein the acquiring the light signal collected by the image sensor to process the light signal to generate a first color space image and a second color space image comprises:

determining the correlation between the first color space image and the second color space image according to the pixel arrangement of the image sensor;

Processing the light signal according to the correlation and interpolation algorithm to obtain the first color space image and the second color space image.
The image processing method according to claim 18, wherein said performing white balance processing on said first color space image according to said first color space image and said second color space image to obtain a white balance processed image comprises :

Preprocessing the first color space image to obtain a first preprocessing image, and processing the second color space image through a virtual channel to obtain a second preprocessing image;

Summarizing the second pre-processed image to the first pre-processed image to perform white balance processing on the first pre-processed image to obtain a white balance processed image.
The image processing method according to claim 20, wherein the preprocessing includes black level compensation processing, lens correction processing and/or dead point compensation processing.
The image processing method according to claim 18, wherein the image processing method further comprises:

performing color interpolation processing and color correction processing on the white balance processed image to obtain an output image.
An image processor, wherein the image processor includes:

An acquisition module, the acquisition module is used to acquire the light signal collected by the image sensor to process the light signal to generate a first color space image and a second color space image;

A white balance processing module, configured to perform white balance processing on the first color space image according to the first color space image and the second color space image to obtain a white balance processed image.