CN113452968B

CN113452968B - Image processing method

Info

Publication number: CN113452968B
Application number: CN202010216438.9A
Authority: CN
Inventors: 任青春; 梁秋芳
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2020-03-25
Filing date: 2020-03-25
Publication date: 2022-04-26
Anticipated expiration: 2040-03-25
Also published as: CN113452968A

Abstract

The invention provides an image processing method. Based on the invention, the color filter array sensitive to the secondary color can be selected as the color filter element used for imaging, and for the target pixel reflecting the secondary color in the image, the color components of the three primary colors related in the pixel value of the target pixel can be obtained based on interpolation operation by using the sampling pixel value of the pixel set defined by the detection window. Compared with the traditional color filter array sensitive to the three primary colors, the color filter array sensitive to the secondary color has higher light sensitivity, so that the light sensitivity loss of an imaging module can be reduced, and the image can contain color information with wider wavelength range, so that the color components of the three primary colors obtained by sampling the pixel set of the detection window based on the secondary color sensitivity and performing interpolation calculation can have higher brightness.

Description

Image processing method

Technical Field

The present invention relates to an image processing technique, and more particularly, to an image processing method for performing demosaicing (demosaicing) processing on an image obtained based on a Color Filter Array (CFA) and a camera using the image processing method.

Background

Color filter arrays are commonly used in color imaging of cameras. In existing color filter arrays, color filters sensitive to different primary colors R (red), G (green), B (blue) are arranged in an array.

When the color filter array covers the photosensitive surface of the imaging module and filters the visible light transmitted on the photosensitive surface, the RGB three-color components in the visible light can be selectively transmitted on the photosensitive surface in a mosaic distribution mode. Thus, in an image filtered based on the color filter array, each pixel approaches to reflect the visible light intensity of only one of the RGB three primary colors, so that the RGB three primary colors are embodied in the image in a mosaic distribution, thereby enabling color information to be included in the image.

By performing demosaicing processing on the image, each pixel can be made to contain full-color information of three primary colors of RGB, thereby obtaining a color image.

However, the wavelength bandwidth of the RGB three primary colors is limited, which results in insufficient sensitivity of the color filter array sensitive to the RGB three primary colors, and thus causes a photosensitive loss to the imaging module, which in turn causes a brightness loss of the image.

Disclosure of Invention

In view of the above, embodiments of the present invention respectively provide an image processing method and a camera using the image processing method, which are helpful for reducing the loss of image brightness due to the light sensing loss of the imaging module.

In one embodiment, there is provided an image processing method including:

acquiring an image obtained by imaging after filtering by a color filter element, wherein the color filter element comprises a color filter array which is arranged in a vertically and horizontally copying manner, the color filter array has a 2 x 2 pixel array specification, the color filter array comprises two first color filters which are sensitive to a first secondary color, a second color filter which is sensitive to a second secondary color and a third color filter which is sensitive to a third secondary color, and the first secondary color, the second secondary color and the third secondary color are respectively formed by combining and mixing different double primary colors in a first primary color, a second primary color and a third primary color;

in response to a target pixel being located in the image, locating a detection window centered on the target pixel, wherein the detection window has a larger pixel array specification than the color filter array;

determining a first pixel subset associated with a first secondary color, a second pixel subset associated with a second secondary color and a third pixel subset associated with a third secondary color in the pixel set in the detection window according to the alignment relationship of each pixel in the pixel set with the first color filter, the second color filter and the third color filter, wherein the target pixel is included in one of the first pixel subset, the second pixel subset and the third pixel subset;

determining a pixel mean of the first set of pixel values, a pixel mean of the second subset of pixel values, and a pixel mean of the third subset of pixel values as a first sampled pixel value associated with the first secondary color, a second sampled pixel value associated with the second secondary color, and a third sampled pixel value associated with the third secondary color, respectively;

determining one of the first primary color, the second primary color, and the third primary color as a preferential interpolation color according to the number of subset pixels respectively included in the first pixel subset, the second pixel subset, and the third pixel subset;

calculating one component of the first color component, the second color component and the third color component of the target pixel, which is associated with the priority interpolation color, by using the first sampling pixel value, the second sampling pixel value and the third sampling pixel value;

and calculating to obtain color components respectively associated with the other two primary colors except the priority interpolation color in the first color component, the second color component and the third color component by utilizing one component of the associated priority interpolation color and corresponding pixel values in the first sampling pixel value, the second sampling pixel value and the third sampling pixel value.

Optionally, determining one of the first primary color, the second primary color, and the third primary color as a preferential interpolation color according to the number of subset pixels respectively included in the first pixel subset, the second pixel subset, and the third pixel subset, includes:

when the number of the subset pixels of the first pixel subset is more than the number of the subset pixels of the second pixel subset and the number of the subset pixels of the third pixel subset, determining the intersection primary color of the second secondary color and the third secondary color as a priority interpolation color;

when the target pixel is aligned with the second color filter and included in the second pixel subset, the number of sub-pixels of the first pixel subset is greater than that of the second pixel subset and is the same as that of the third pixel subset, and the intersection primary color of the first secondary color and the third secondary color is determined as a preferential interpolation color;

when the target pixel is aligned with the third color filter and included in the third pixel subset, the number of sub-pixels of the first pixel subset is the same as the number of sub-pixels of the second pixel subset and is greater than the number of sub-pixels of the third pixel subset, and the intersection primary color of the first secondary color and the second secondary color is determined as a preferential interpolation color.

Optionally, locating the target pixel in the image comprises: any pixel in an effective pixel area surrounded by the redundant edge pixels of the image is determined as a target pixel.

Optionally, the peak wavelength of the first primary color is in a range of 590nm to 640 nm; the peak wavelength of the second primary color is within the range of 500 nm-560 nm; the peak wavelength of the third primary color is in the range of 430nm to 480 nm.

Optionally, the first secondary color is obtained by mixing a first primary color and a third primary color, the second secondary color is obtained by mixing the first primary color and the second primary color, and the third secondary color is obtained by mixing the second primary color and the third primary color.

Optionally, the spectral power distribution of the first secondary color has a continuous valley section in a wavelength range of 480nm to 590nm, and continuous peak sections located at both sides of the continuous valley section; the spectral power distribution of the second secondary color has a continuous valley region in a wavelength region of less than 500nm and a continuous peak region in a wavelength region of 500nm or more; the spectral power of the third secondary color has a peak in a wavelength band of 430nm to 560nm and is attenuated from both sides of the peak.

Optionally, two first color filters are diagonally arranged in the color filter array, and the second color filter and the third color filter are diagonally arranged in the color filter array in mirror image with the first color filters.

Optionally, two first color filters are arranged diagonally in the color filter array at the upper left and lower right corners, respectively, a second color filter array is arranged at the upper right corner of the color filter array, and a third color filter is arranged at the lower left corner of the color filter array.

Optionally, the detection window has a pixel array specification of 3 × 3.

In another embodiment, there is provided a camera including: an imaging module; the color filter element covers the photosensitive surface of the imaging module, and comprises color filter arrays in longitudinal and transverse copying arrangement, each color filter array comprises two first color filters sensitive to a first secondary color, a second color filter sensitive to a second secondary color and a third color filter sensitive to a third secondary color, and the first secondary color, the second secondary color and the third secondary color are formed by combining and mixing different double primary colors in a first primary color, a second primary color and a third primary color respectively; and a processor for performing the image processing method as described above.

In another embodiment, there is provided another camera including:

an optical lens group for penetrating ambient light into the interior of the camera;

a color filter array having secondary color filters, wherein the secondary color filters include cyan, yellow, and magenta filters, and wherein a ratio of the number of cyan filters, the number of yellow filters, and the number of magenta filters in the color filter array is 1:1: 2; wherein the cyan filter is configured to simultaneously transmit blue and green light of the ambient light, the yellow filter is configured to simultaneously transmit red and green light of the ambient light, and the magenta filter is configured to simultaneously transmit blue and red light of the ambient light;

an image sensor for sensing light transmitted through the secondary color filters of the color filter array and generating a secondary color image;

and the image processor is used for generating an RGB three-primary-color image by processing the secondary color image according to the preset color mixing relation between the secondary color and the RGB three primary colors.

Based on the above embodiment, the color filter array sensitive to the secondary color may be selected as the color filter element used for imaging, and for the target pixel reflecting the secondary color in the image, the color components associated with the three primary colors in the pixel value of the target pixel may be obtained based on interpolation operation by using the sampled pixel values of the pixel set defined by the detection window. Compared with the traditional color filter array sensitive to the three primary colors, the color filter array sensitive to the secondary color has higher light sensitivity, so that the light sensitivity loss of an imaging module can be reduced, and the image can contain color information with wider wavelength range, so that the color components of the three primary colors obtained by sampling the pixel set of the detection window based on the secondary color sensitivity and performing interpolation calculation can have higher brightness.

Drawings

The following drawings are only schematic illustrations and explanations of the present invention, and do not limit the scope of the present invention:

FIG. 1 is a schematic diagram of an exemplary configuration of a camera in one embodiment;

FIG. 2 is a schematic spectral diagram of the sensitive secondary colors of the secondary color filter array used in the camera of FIG. 1;

FIG. 3 is a schematic diagram of a first example of a secondary color filter array for use with the camera of FIG. 1;

FIG. 4 is a schematic diagram of a second example of a secondary color filter array for use with the camera of FIG. 1;

FIG. 5 is a schematic illustration of the sensitivity response band of a secondary color filter array used in the camera of FIG. 1;

FIG. 6 is a schematic flow diagram illustrating an exemplary method of image processing to support secondary color filter array imaging in one embodiment;

FIG. 7 is a schematic view of an extended flow of the image processing method shown in FIG. 6 based on window mean sampling;

FIG. 8 is a schematic diagram of an extended flow of the image processing method shown in FIG. 7 based on base color step interpolation;

FIGS. 9a to 9d are schematic diagrams of an example process of demosaicing based on the first example of the secondary color filter array shown in FIG. 3 to implement the flow shown in FIG. 8;

fig. 10a to 10d are schematic diagrams of an example process of demosaicing based on the second example of the secondary color filter array shown in fig. 4 to implement the flow shown in fig. 8.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and examples.

Fig. 1 is an exemplary configuration diagram of a video camera in one embodiment. Referring to fig. 1, in one embodiment, the camera may include:

an imaging module 110;

a color filter 120, wherein the color filter 120 can cover the photosensitive surface of the imaging module 110; and the number of the first and second groups,

a processor 130, wherein the processor 130 is configured to perform image processing including demosaicing processing on the image output by the imaging module 110.

As can be seen in fig. 1, the color filter element 120 includes a color filter array 100 in a vertically and horizontally replicated arrangement. The Color filter array has a 2 × 2 pixel array specification, and the Color filter array 100 may be a Secondary Color (Secondary Color) array. The secondary color filter array 100 may include two first color filters 10a sensitive to the first secondary color SC1, one second color filter 10b sensitive to the second secondary color SC2, and one third color filter 10c sensitive to the third secondary color SC3, and the first secondary color SC1, the second secondary color SC2, and the third secondary color SC3 are respectively formed by mixing different two primary colors of the first primary color, the second primary color, and the third primary color of the RGB three primary colors.

Fig. 2 is a schematic diagram of the spectrum of the sensitive secondary color of the secondary color filter array used in the camera of fig. 1. Referring to fig. 2, the RGB three primary colors include red R with a peak wavelength in a range of 590nm to 640nm, green G with a peak wavelength in a range of 500nm to 560nm, and blue B with a peak wavelength in a range of 430nm to 480nm, and accordingly, the color selection ranges of the first secondary color SC1, the second secondary color SC2, and the third secondary color SC3 may include magenta m (magenta) obtained by mixing red and blue, yellow y (yellow) obtained by mixing red and green, and cyan c (cyan) obtained by mixing blue and green, wherein:

the spectral power distribution of magenta M has a continuous valley section in the wavelength range of 480nm to 590nm and continuous peak sections located at both sides of the continuous valley section;

the spectral power distribution of yellow Y has a continuous valley region in a wavelength region of less than 500nm and a continuous peak region in a wavelength region of 500nm or more;

the spectral power of cyan C has a peak in a wavelength band of 430nm to 560nm, and attenuates from both sides of the peak.

Fig. 3 is a schematic diagram of a first example of a secondary color filter array for use with the camera of fig. 1. Referring to fig. 3, in the first example, the first secondary color SC1 is magenta M, the second secondary color SC2 is yellow Y, and the third secondary color SC3 is cyan C, in which case, the two first color filters 10a ' are color filters FM sensitive to magenta M, the second color filter 10b ' is a color filter FY sensitive to yellow Y, and the third color filter 10C ' is a color filter FC sensitive to cyan C, and the two first color filters 10a ' (FM) are diagonally arranged in the color filter array 100 ', and the second color filter 10b ' (FY) and the third color filter 10C ' (FC) are diagonally arranged in the color filter array 100 ' in mirror image with the first color filter 10a ' (FM).

In fig. 3, with two first color filters 10a '(FM) in an upper left and lower right diagonal arrangement, and a second color filter 10 b' (FY) and a third color filter 10c '(FC) in an upper right and lower left diagonal arrangement, the color filter array 100' may be referred to as a MYCM color filter array.

Of course, the arrangement orientation of the color filter array 100' in fig. 3 may also be mirror-adjusted, and the arrangement positions of the second color filter FY and the third color filter FC in fig. 3 may also be reversed.

Fig. 4 is a schematic diagram of a second example of a secondary color filter array for use with the camera of fig. 1. Referring to fig. 4, in the second example, the first secondary color SC1 is yellow Y, the second secondary color SC2 is cyan C, and the third secondary color SC3 is magenta M, in which case, the two first color filters 10a "are color filters FY sensitive to yellow Y, the second color filter 10 b" is color filter FC sensitive to cyan C, and the third color filter 10C "is color filter FM sensitive to magenta M, and the two first color filters 10 a" (FY) are diagonally arranged in the color filter array 100 ", and the second color filter 10 b" (FC) and the third color filter 10C "(FM) are diagonally arranged in the color filter array 100" in mirror image with the first color filter 10a "(FY).

In fig. 4, with two first color filters 10a "(FY) arranged diagonally upper right and lower left, and a second color filter 10 b" (FC) and a third color filter 10c "(FM) arranged diagonally upper left and lower right, the color filter array 100" may be referred to as a CYYM color filter array.

Of course, the arrangement orientation of the color filter array 100 "in fig. 4 may be adjusted in a mirror image, and the diagonal arrangement positions of the second color filter FC and the third color filter FM in fig. 4 may be changed.

The above first and second examples are intended to illustrate what color selection schemes may not be unique for the first secondary color SC1, the second secondary color SC2, and the third secondary color SC3 from among magenta M, yellow Y, and cyan C.

Fig. 5 is a schematic illustration of the sensitivity response band of a secondary color filter array used in the camera of fig. 1. Referring to fig. 5, the sensitive response band of the secondary color filter sensitive to magenta M (transmitting magenta M) can be considered as the wavelength band combination of red R and blue B; a secondary color filter sensitive to yellow Y (transmitting yellow Y), whose sensitive response band can be regarded as a wavelength band combination of red R and green G; the sensitive response band of the secondary color filter sensitive to cyan C (transmitting cyan C) can be considered as a wavelength band combination of green G and blue B. Therefore, compared with the conventional color filter array sensitive to three primary colors, no matter what color selection scheme is adopted by the first secondary color SC1, the second secondary color SC2 and the third secondary color SC3 from the magenta color M, the yellow color Y and the cyan color C, the color filter array sensitive to the secondary color in the embodiment can have higher sensitivity performance, so that the light sensitivity loss of the imaging module can be reduced, and the image can contain color information with wider wavelength range.

That is, the camera as shown in FIG. 1 may include an optical lens group (not shown in FIG. 1) for penetrating ambient light into the interior of the camera. Also, the camera shown in fig. 1 may further include:

a color filter array 100 having

secondary color filters

10a, 10b, and 10c, wherein the

secondary color filters

10a, 10b, and 10c include cyan color filters (

third color filters

10c or 10c 'or 10c "), yellow color filters (

second color filters

10b or 10 b' or 10 b"), and magenta color filters (

first color filters

10a or 10a 'or 10a "), than a ratio of the number of the cyan color filters (

third color filters

10c or 10 c' or 10 c"), the number of the yellow color filters (

second color filters

10b or 10b 'or 10b "), and the number of the magenta color filters (

first color filters

10a or 10 a' or 10 a") in the color filter array 100 may be 1:1: 2;

wherein, a cyan color filter (the

third color filter

10c or 10c ' or 10c ") is used to simultaneously transmit light of blue B and green G in ambient light, a yellow color filter (the second color filter 10B or 10B ' or 10B") is used to simultaneously transmit light of red R and green G in ambient light, and a magenta color filter (the

first color filter

10a or 10a ' or 10a ") is used to simultaneously transmit light of blue B and red R in ambient light;

an imaging module 110 (e.g., an image sensor) for sensing the light transmitted through the

secondary color filters

10a, 10b, and 10c of the color filter array 10 and generating a secondary color image;

and a processor 130 (e.g., an image sensor) for generating an RGB three primary color image by processing the secondary color image according to a preset color mixing relationship of the secondary color and the RGB three primary colors.

However, unlike the image formed by imaging after filtering with the conventional color filter array, in the image formed by filtering with the secondary color filter array, each pixel is associated with a corresponding secondary color, and in this case, how to better utilize the secondary colors associated with each pixel in the image to obtain the color components of the RGB three primary colors of the pixel becomes a problem to be further solved.

To this end, in the embodiments described below, an image processing method compatible with different color selection schemes of the first secondary color SC1, the second secondary color SC2, and the third secondary color SC3 from among magenta color M, yellow color Y, and cyan color C is provided to obtain color components of RGB three primary colors of a pixel by performing demosaicing processing on an image including secondary color pixels.

FIG. 6 is a schematic flow diagram illustrating an exemplary method of image processing to support secondary color filter array imaging in one embodiment. Referring to fig. 6, the image processing method in this embodiment may include:

s610: the method comprises the steps of obtaining an image obtained by imaging after filtering by a color filter element, wherein the color filter element comprises a color filter array which is arranged in a vertically and horizontally copying mode, the color filter array has a 2 x 2 pixel array specification, the color filter array comprises two first color filters which are sensitive to a first secondary color, a second color filter which is sensitive to a second secondary color and a third color filter which is sensitive to a third secondary color, and the first secondary color, the second secondary color and the third secondary color are formed by combining and mixing different double primary colors in a first primary color, a second primary color and a third primary color respectively.

S630: when a target pixel is located in the image, a detection window is located centered on the target pixel, wherein the detection window has a larger pixel array specification than the color filter array.

S650: first sampled pixel values associated with a first secondary color, second sampled pixel values associated with a second secondary color, and third sampled pixel values associated with a third secondary color are obtained from a set of pixels within a detection window.

S670: and obtaining a first color component related to a first base color, a second color component related to a second base color and a third color component related to a third base color in the pixel values of the target pixel based on interpolation calculation by using the obtained first sampling pixel value, the obtained second sampling pixel value and the obtained third sampling pixel value.

Based on the above-mentioned flow, a first color component associated with the first base color, a second color component associated with the second base color, and a third color component associated with the third base color in the pixel values of the target pixel may be interpolated by using the pixel values of the pixel set within the detection window.

For the sampling in the pixel set in the detection window, an average sampling scheme may be adopted to obtain a sampling value capable of representing the secondary color characteristic of the local region in a limited neighborhood pixel range of the target pixel surrounded by the detection window.

Fig. 7 is an expanded flow diagram of the image processing method shown in fig. 6 based on window mean sampling. Referring to fig. 7, the image processing method in this embodiment may be expanded to include the following steps:

s710: the method comprises the steps of obtaining an image obtained by imaging after filtering by a color filter element, wherein the color filter element comprises a color filter array which is arranged in a vertically and horizontally copying mode, the color filter array has a 2 x 2 pixel array specification, the color filter array comprises two first color filters which are sensitive to a first secondary color, a second color filter which is sensitive to a second secondary color and a third color filter which is sensitive to a third secondary color, and the first secondary color, the second secondary color and the third secondary color are formed by combining and mixing different double primary colors in a first primary color, a second primary color and a third primary color respectively.

S730: when a target pixel is located in the image, a detection window is located centered on the target pixel, wherein the detection window has a larger pixel array specification than the color filter array.

S751: and determining a first pixel subset associated with the first secondary color, a second pixel subset associated with the second secondary color and a third pixel subset associated with the third secondary color in the pixel set according to the alignment relation of each pixel in the pixel set and the first color filter, the second color filter and the third color filter, wherein the target pixel is included in one of the first pixel subset, the second pixel subset and the third pixel subset.

S753: and determining the pixel mean value of the first pixel value set, the pixel mean value of the second pixel value subset and the pixel mean value of the third pixel value subset as a first sampling pixel value, a second sampling pixel value and a third sampling pixel value respectively.

S770: and obtaining a first color component related to a first base color, a second color component related to a second base color and a third color component related to a third base color in the pixel values of the target pixel based on interpolation calculation by using the obtained first sampling pixel value, the obtained second sampling pixel value and the obtained third sampling pixel value.

Since the proportion of each pixel subset associated with different secondary colors in the pixel set may be different, the sample cardinalities of the first sampled pixel value, the second sampled pixel value, and the third sampled pixel value, which are the same pixel mean, are also different. The interpolation result of the color components of the three primary colors can be influenced by the difference of the sample cardinality. For example, for a sampled pixel value with too few sample bases, the color component interpolation effect of the primary color with the color mixing relationship of the secondary color associated with the sampled pixel value is weaker; for the sampling pixel value with more sample base numbers, the stronger the interpolation effect of the color component of the primary color with the color mixing relation of the secondary color associated with the sampling pixel value is.

For this, the interpolation may be preferentially performed by selecting one of the first, second, and third primaries which has a relatively significant influence on the interpolation effect, in combination with the color mixing relationship and the sample cardinality.

Fig. 8 is an expanded flow diagram of the image processing method shown in fig. 7 based on the base color step interpolation. Referring to fig. 8, the image processing method in this embodiment may be expanded to include the following steps:

s810, S830, S851 and S853 substantially the same as S710, S730, S751 and S753 in the flow shown in fig. 7, and

s871: and counting the number of subset pixels respectively included in the first pixel subset, the second pixel subset and the third pixel subset.

S873: one of the first primary color, the second primary color, and the third primary color is determined as a priority interpolation color according to the determined number of the subset pixels. For example:

when the number of the subset pixels of the first pixel subset is greater than the number of the subset pixels of the second pixel subset and the number of the subset pixels of the third pixel subset, the sample cardinality of the second sampled pixel value associated with the second secondary color and the third sampled pixel value associated with the third secondary color is relatively small, and at this time, the interpolation effect of the intersection primary color (one of the second primary color and the third primary color which has no color mixing relation with the first secondary color) of the second secondary color and the third secondary color is relatively weak, so that the intersection primary color of the second secondary color and the third secondary color can be determined as a priority interpolation color to reinforce the color component interpolation effect of the intersection primary color;

when the number of sub-set pixels of the first pixel subset is greater than that of the second pixel subset and is the same as that of the third pixel subset, the sample cardinality of the first sampled pixel value associated with the first secondary color and the third sampled pixel value associated with the third secondary color is relatively high, and thus the intersection primary color of the first secondary color and the third secondary color can be determined as a preferential interpolation color to maximize the color component interpolation effect of the intersection primary color;

when the number of sub-set pixels of the first pixel subset is the same as the number of sub-set pixels of the second pixel subset and is greater than the number of sub-set pixels of the third pixel subset, the sample cardinality of the first sampled pixel value associated with the first secondary color and the second sampled pixel value associated with the second secondary color is relatively high, and thus the intersection primary color of the first secondary color and the second secondary color may be determined as a preferential interpolation color to maximize the color component interpolation effect of the intersection primary color.

S875: and calculating one component of the first color component, the second color component and the third color component of the target pixel, which is associated with the priority interpolation color, by using the first sampling pixel value, the second sampling pixel value and the third sampling pixel value.

S877: and calculating to obtain color components respectively associated with the other two primary colors except the priority interpolation color in the first color component, the second color component and the third color component by utilizing one component of the associated priority interpolation color and corresponding pixel values in the first sampling pixel value, the second sampling pixel value and the third sampling pixel value.

That is, the color components respectively associated with the other two primary colors are calculated with the priority interpolation color, which is reinforced or maximized by the interpolation effect, as a reference.

For a better understanding of the flow shown in fig. 8, the following is illustrated with reference to an example.

Fig. 9a to 9d are schematic diagrams of an example process of demosaicing based on the first example of the secondary color filter array shown in fig. 3 to implement the flow shown in fig. 8. Referring first to fig. 9a, in an example process of implementing demosaicing based on the MYCM color filter array 100' as shown in fig. 3, the detection window 900 is set to have a 3 × 3 pixel array specification, and: in an image formed by filtering with the color filter element (MYCM filter array 100 '), pixel arrangement respectively associated with the first secondary color M, the second secondary color Y, and the third secondary color C is the same as the arrangement of each of the color filters FM, FY, and FC in the MYCM filter array 100'.

Please refer to fig. 9b after referring to fig. 9 a:

at S911, when the target pixel M0 associated with the first secondary color M is located in the image, the detection window 900 of the 3 × 3 pixel array specification is located centered around the target pixel M0;

at S913, according to the alignment relationship of the secondary color filters in the pixel Set, a first pixel subset Set _ M ═ { M0, M1, M2, M3, M4}, a second pixel subset Set _ Y ═ { Y1, Y2} associated with the second secondary color Y, and a third pixel subset Set _ C ═ { C1, C2} associated with the third secondary color C in the pixel Set in the detection window 900 are determined, wherein the target pixel M0 is included in the first pixel subset Set _ M.

At S915, the pixel mean Avg _ M of the first Set of pixel values Set _ M, the pixel mean Avg _ Y of the second subset of pixel values Set _ Y, and the pixel mean Avg _ C of the third subset of pixel values Set _ C are respectively determined as a first sampled pixel value Samp _ M, a second sampled pixel value Samp _ Y, and a third sampled pixel value Samp _ C, that is:

the first sample pixel value Samp _ M ═ Avg _ M ═ (M0+ M1+ M2+ M3+ M4)/5;

the second sample pixel value Samp _ Y ═ Avg _ Y ═ Y1+ Y2)/2;

the third sample pixel value Samp _ C ═ Avg _ C ═ C1+ C2)/2.

At S917, the numbers of sub-pixels respectively included in the first pixel subset Set _ M, the second pixel subset Set _ Y, and the third pixel subset Set _ C are counted, and since the number (5) of sub-pixels of the first pixel subset Set _ M is greater than the number (2) of sub-pixels of the second pixel subset Set _ Y and the number (2) of sub-pixels of the third pixel subset Set _ C, the primary color G of the intersection of the second secondary color Y and the third secondary color C may be determined as a preferential interpolation color according to the determined numbers of sub-pixels, so as to reinforce the color component interpolation effect of the primary color G, with reference to the color blending relationship between the secondary colors and the primary colors.

Then, using the first sampled pixel value Samp _ M, the second sampled pixel value Samp _ Y, and the third sampled pixel value Samp _ C, the second color component Comp _ G of the target pixel M0 associated with the priority interpolation color G is calculated, that is, based on the color mixture relationship of M ═ R + B and Y + C ═ R + B +2G, the following is calculated:

Comp_G＝(Samp_Y+Samp_C-Samp_M)/2。

then, using one component Comp _ G of the associated priority interpolation color G, the second sampled pixel value Samp _ Y, and the third sampled pixel value Samp _ C, the color components Comp _ R and Comp _ B respectively associated with the other two primaries R and B except the priority interpolation color G are calculated, that is, according to the color mixture relationship of Y ═ R + G and C ═ B + G, the following calculation results:

Comp_R＝Samp_Y-Comp_G；

Comp_B＝Samp_C-Comp_G。

thus, the color components { Comp _ R, Comp _ G, Comp _ B of the three primary colors for the pair of the target pixel M0 can be obtained.

Please refer to fig. 9c after referring to fig. 9 a:

at S921, when a target pixel Y0 associated with the second secondary color Y is located in the image, the detection window 900 of the 3 × 3 pixel array specification is located centering on the target pixel Y0;

at S923, according to the alignment relationship of the secondary color filter of the pixel Set, a first pixel subset Set _ M ═ { M1, M2, M3, M4} associated with the first secondary color M, a second pixel subset Set _ Y ═ { Y0} associated with the second secondary color Y, and a third pixel subset Set _ C ═ { C1, C2, C3, C4} associated with the third secondary color C in the pixel Set in the detection window 900 are determined, wherein the target pixel Y0 is included in the third pixel subset Set _ Y.

At S925, the pixel mean Avg _ M of the first Set of pixel values Set _ M, the pixel mean Avg _ Y of the second subset of pixel values Set _ Y, and the pixel mean Avg _ C of the third subset of pixel values Set _ C are determined as a first sampled pixel value Samp _ M, a second sampled pixel value Samp _ Y, and a third sampled pixel value Samp _ C, respectively, that is:

the first sample pixel value Samp _ M ═ Avg _ M ═ (M1+ M2+ M3+ M4)/4;

the second sample pixel value Samp _ Y ═ Avg _ Y ═ Y0;

the second sample pixel value Samp _ C ═ Avg _ C ═ (C1+ C2+ C3+ C4)/4.

At S927, the numbers of the sub-Set pixels respectively included in the first pixel subset Set _ M, the second pixel subset Set _ Y, and the third pixel subset Set _ C are counted, and since the number (4) of the sub-Set pixels of the first pixel subset Set _ M is greater than the number (1) of the sub-Set pixels of the second pixel subset Set _ Y and is the same as the number (4) of the sub-Set pixels of the third pixel subset Set _ C, the intersection primary color B of the first secondary color M and the third secondary color C may be determined as a priority interpolation color according to the determined number of the sub-Set pixels, so as to maximize the color component interpolation effect of the intersection primary color B, with reference to the color blending relationship between the secondary colors and the primary colors.

Then, the third color component Comp _ B of the target pixel Y0 associated with the priority interpolation color B is calculated using the first sampled pixel value Samp _ M, the second sampled pixel value Samp _ Y, and the third sampled pixel value Samp _ C, that is, based on the color mixture relationship of M + C ═ R +2B + G and Y ═ R + G:

Comp_B＝(Samp_M+Samp_C-Samp_Y)/2。

then, using one component Comp _ B associated with the priority interpolation color B, the first sampled pixel value Samp _ M, and the third sampled pixel value Samp _ C, color components Comp _ R and Comp _ G associated with the other two primaries R and G, respectively, except for the priority interpolation color B are calculated, that is, according to the color mixture relationship of M ═ R + B and C ═ B + G, the following calculation results:

Comp_R＝Samp_M-Comp_B；

Comp_G＝Samp_C-Comp_B。

thus, the color components { Comp _ R, Comp _ G, Comp _ B of the three primary colors for the pair of the target pixel Y0 can be obtained.

Please refer to fig. 9d after referring to fig. 9 a:

at S931, when the target pixel C0 associated with the third secondary color C is located in the image, the detection window 900 of the 3 × 3 pixel array specification is located centering on the target pixel C0;

at S933, according to the alignment relationship of the secondary color filter in the pixel Set, a first pixel subset Set _ M ═ { M1, M2, M3, M4} associated with the first secondary color M, a second pixel subset Set _ Y ═ { Y1, Y2, Y3, Y4} associated with the second secondary color Y, and a third pixel subset Set _ C ═ C { C0} associated with the third secondary color C in the pixel Set in the detection window 900 are determined, wherein the target pixel Y0 is included in the second pixel subset Set _ Y.

At S935, the pixel mean Avg _ M of the first Set of pixel values Set _ M, the pixel mean Avg _ Y of the second subset of pixel values Set _ Y, and the pixel mean Avg _ C of the third subset of pixel values Set _ C are determined as a first sampled pixel value Samp _ M, a second sampled pixel value Samp _ Y, and a third sampled pixel value Samp _ C, respectively, that is:

the first sample pixel value Samp _ M ═ Avg _ M ═ (M1+ M2+ M3+ M4)/4;

the second sample pixel value Samp _ Y ═ Avg _ Y ═ (Y1+ Y2+ Y3+ Y4)/4;

the second sample pixel value Samp _ C ═ Avg _ C ═ C0.

At S937, the numbers of the subset pixels respectively included in the first pixel subset Set _ M, the second pixel subset Set _ Y, and the third pixel subset Set _ C are counted, and since the number (4) of the subset pixels of the first pixel subset Set _ M is greater than the number (1) of the subset pixels of the third pixel subset Set _ C and is the same as the number (4) of the subset pixels of the second pixel subset Set _ Y, the intersection primary color R of the first secondary color M and the second secondary color Y may be determined as a priority interpolation color according to the determined number of the subset pixels with reference to the color blending relationship between the secondary colors and the primary colors to maximize the color component interpolation effect of the intersection primary color R.

Then, the first color component Comp _ R associated with the target pixel C0 and the priority interpolation color R is calculated using the first sample pixel value Samp _ M, the second sample pixel value Samp _ Y, and the third sample pixel value Samp _ C, that is, based on the color mixture relationship of M + Y ═ 2R + B + G and C ═ B + G, the following is calculated:

Comp_R＝(Samp_M+Samp_Y-Samp_C)/2。

then, using one component Comp _ R associated with the priority interpolation color R, and the first sampled pixel value Samp _ M and the second sampled pixel value Samp _ Y, color components Comp _ B and Comp _ G associated with the other two primaries B and G except the priority interpolation color R are calculated, that is, according to the color mixture relationship of M ═ R + B and Y ═ R + G, the following calculation results:

Comp_G＝Samp_Y-Comp_R；

Comp_B＝Samp_M-Comp_R。

thus, the color components { Comp _ R, Comp _ G, Comp _ B of the three primary colors for the pair of the target pixel C0 can be obtained.

In addition, since the pixel at the edge of the image cannot be the positioning center of the detection window 900, there may be a redundant edge pixel in the image imaged after being filtered by the color filter element (MYCM filter array 100'), and accordingly, for the target pixel M0 or Y0 or C0, any pixel in the effective pixel region surrounded by the redundant edge pixel may be determined as the target pixel. When the mosaic processing is completed, the redundant edge pixels may be clipped or other processing performed.

Fig. 10a to 10d are schematic diagrams of an example process of demosaicing based on the second example of the secondary color filter array shown in fig. 4 to implement the flow shown in fig. 8. Referring first to fig. 10a, in an example process of implementing demosaicing based on the CYYM filter 100 "array as shown in fig. 4, a detection window 910 is set to have a 3 × 3 pixel array specification, and: in an image formed by filtering with the color filter element (CYYM filter array 100 "), pixel arrangements respectively associated with the first secondary color Y, the second secondary color C, and the third secondary color M are the same as the arrangement of the respective color filters FY, FC, FM in the CYYM filter array 100 ″.

Please refer to fig. 10b after referring to fig. 10 a:

at S951, when a target pixel Y0 associated with the first secondary color Y is located in the image, a detection window 910 of a 3 × 3 pixel array size is located centering on the target pixel Y0;

at S953, according to the alignment relationship of the secondary color filter in the pixel Set, a first pixel subset Set _ Y ═ { Y0, Y1, Y2, Y3, Y4} associated with the first secondary color Y, a second pixel subset Set _ C ═ { C1, C2} associated with the second secondary color C, and a third pixel subset Set _ M ═ M { M1, M2} associated with the third secondary color M in the pixel Set in the detection window 910 are determined, wherein the target pixel Y0 is included in the first pixel subset Set _ Y.

At S955, the pixel mean Avg _ Y of the first Set of pixel values Set _ Y, the pixel mean Avg _ C of the second subset of pixel values Set _ C, and the pixel mean Avg _ M of the third subset of pixel values Set _ M are respectively determined as a first sampled pixel value Samp _ Y, a second sampled pixel value Samp _ C, and a third sampled pixel value Samp _ M, that is:

the first sample pixel value Samp _ Y ═ Avg _ Y ═ (Y0+ Y1+ Y2+ Y3+ Y4)/5;

the second sample pixel value Samp _ C ═ Avg _ C ═ C1+ C2)/2;

the third sample pixel value Samp _ M ═ Avg _ M ═ M1+ M2)/2.

At S957, the numbers of the sub-pixels respectively included in the first pixel subset Set _ Y, the second pixel subset Set _ C, and the third pixel subset Set _ M are counted, and since the number (5) of the sub-pixels of the first pixel subset Set _ Y is greater than the number (2) of the sub-pixels of the second pixel subset Set _ C and the number (2) of the sub-pixels of the third pixel subset Set _ M, the intersection primary color B of the second secondary color C and the third secondary color M may be determined as a priority interpolation color according to the determined numbers of the sub-pixels, so as to reinforce the color component interpolation effect of the intersection primary color B, with reference to the color blending relationship between the secondary colors and the primary colors.

Then, the third color component Comp _ B associated with the target pixel Y0 and the priority interpolation color B is calculated using the first sampled pixel value Samp _ Y, the second sampled pixel value Samp _ C, and the third sampled pixel value Samp _ M, that is, based on the color mixture relationship of Y ═ R + G and M + C ═ R +2B + G:

Comp_B＝(Samp_M+Samp_C-Samp_Y)/2。

then, using one component Comp _ B associated with the priority interpolation color B, the second sampled pixel value Samp _ C, and the third sampled pixel value Samp _ M, color components Comp _ R and Comp _ G associated with the other two primaries R and G, respectively, except for the priority interpolation color B are calculated, that is, according to the color mixture relationship of M ═ R + B and C ═ B + G, the following calculation results:

Comp_R＝Samp_M-Comp_B；

Comp_G＝Samp_C-Comp_B。

Please refer to fig. 10c after referring to fig. 10 a:

at S961, when the target pixel C0 associated with the second secondary color C is located in the image, the detection window 910 of the 3 × 3 pixel array specification is located centering on the target pixel C0;

at S963, according to the alignment relationship of the secondary color filter of the pixel Set, a first pixel subset Set _ Y ═ { Y1, Y2, Y3, Y4} associated with the first secondary color Y, a second pixel subset Set _ C ═ { C0} associated with the second secondary color C, and a third pixel subset Set _ M ═ { M1, M2, M3, M4} associated with the third secondary color M in the pixel Set in the detection window 900 are determined, wherein the target pixel C0 is included in the second pixel subset Set _ C.

At S965, the pixel mean Avg _ Y of the first Set of pixel values Set _ Y, the pixel mean Avg _ C of the second subset of pixel values Set _ C, and the pixel mean Avg _ M of the third subset of pixel values Set _ M are determined as a first sampled pixel value Samp _ Y, a second sampled pixel value Samp _ C, and a third sampled pixel value Samp _ M, respectively, that is:

the first sample pixel value Samp _ Y ═ Avg _ Y ═ (Y1+ Y2+ Y3+ Y4)/4;

the second sample pixel value Samp _ C ═ Avg _ C ═ Y0;

the second sampled pixel value Samp _ M ═ Avg _ M ═ (M1+ M2+ M3+ M4)/4.

At S967, the numbers of the sub-pixels respectively included in the first pixel subset Set _ Y, the second pixel subset Set _ C, and the third pixel subset Set _ M are counted, and since the number (4) of the sub-pixels of the first pixel subset Set _ Y is greater than the number (1) of the sub-pixels of the second pixel subset Set _ C and is the same as the number (4) of the sub-pixels of the third pixel subset Set _ M, the intersection primary color R of the first secondary color Y and the third secondary color M may be determined as a priority interpolation color according to the determined number of the sub-pixels, so as to maximize the color component interpolation effect of the intersection primary color R, with reference to the color blending relationship between the secondary colors and the primary colors.

Then, the first color component Comp _ R associated with the target pixel C0 and the priority interpolation color R is calculated using the first sample pixel value Samp _ Y, the second sample pixel value Samp _ C, and the third sample pixel value Samp _ M, that is, based on the color mixture relationship of M + Y ═ 2R + B + G and C ═ B + G, the following is calculated:

Comp_R＝(Samp_Y+Samp_M-Samp_C)/2。

then, using one component Comp _ R of the associated priority interpolation color R, the first sampled pixel value Samp _ Y, and the third sampled pixel value Samp _ M, color components Comp _ G and Comp _ B respectively associated with the other two primaries G and B except the priority interpolation color R are calculated, that is, according to the color mixture relationship of Y ═ R + G and M ═ R + B, the following calculation results:

Comp_G＝Samp_Y-Comp_R；

Comp_B＝Samp_M-Comp_R。

Please refer to fig. 10d after referring to fig. 10 a:

at S971, when the target pixel M0 associated with the third secondary color M is located in the image, the detection window 910 of the 3 × 3 pixel array specification is located centered on the target pixel M0;

at S973, according to the alignment relationship of the secondary color filter of the pixel Set, a first pixel subset Set _ Y ═ { Y1, Y2, Y3, Y4} associated with the first secondary color Y, a second pixel subset Set _ C ═ { C1, C2, C3, C4} associated with the second secondary color C, and a third pixel subset Set _ M ═ M { M0} associated with the third secondary color M in the pixel Set within the detection window 910 are determined, wherein the target pixel M0 is included in the third pixel subset Set _ M.

At S975, the pixel mean Avg _ Y of the first Set of pixel values Set _ Y, the pixel mean Avg _ C of the second subset of pixel values Set _ C, and the pixel mean Avg _ M of the third subset of pixel values Set _ M are determined as a first sampled pixel value Samp _ Y, a second sampled pixel value Samp _ C, and a third sampled pixel value Samp _ M, respectively, that is:

the first sample pixel value Samp _ Y ═ Avg _ Y ═ (Y1+ Y2+ Y3+ Y4)/4;

the second sample pixel value Samp _ C ═ Avg _ C ═ C1+ C2+ C3+ C4)/4;

the third sample pixel value Samp _ M ═ Avg _ M ═ M0.

At S977, the numbers of the sub-pixels respectively included in the first pixel subset Set _ Y, the second pixel subset Set _ C, and the third pixel subset Set _ M are counted, and since the number (4) of the sub-pixels of the first pixel subset Set _ Y is greater than the number (1) of the sub-pixels of the third pixel subset Set _ M and is the same as the number (4) of the sub-pixels of the second pixel subset Set _ C, the primary color G of the first secondary color Y and the second secondary color C may be determined as a priority interpolation color according to the determined number of the sub-pixels with reference to the color blending relationship between the secondary colors and the primary colors to maximize the color component interpolation effect of the intersection primary color G.

Then, using the first sampled pixel value Samp _ Y, the second sampled pixel value Samp _ C, and the third sampled pixel value Samp _ M, the second color component Comp _ G of the target pixel M0 associated with the priority interpolation color G is calculated, that is, based on the color mixture relationship of Y + Y ═ R +2G + B and M ═ R + B:

Comp_G＝(Samp_Y+Samp_C-Samp_M)/2。

then, using one component Comp _ G of the associated priority interpolation color G, and the first sampled pixel value Samp _ Y and the second sampled pixel value Samp _ C, color components Comp _ R and Comp _ B respectively associated with the other two primaries R and B except the priority interpolation color G are calculated, that is, according to the color mixture relationship of Y ═ R + G and C ═ B + G, the following calculation results:

Comp_R＝Samp_Y-Comp_G；

Comp_B＝Samp_C-Comp_G。

In addition, since the pixel at the edge of the image cannot be the positioning center of the detection window 910, there may be a redundant edge pixel in the image imaged after being filtered by the color filter element (CYYM color filter array 100 "), and accordingly, for the target pixel Y0 or C0 or M0, any pixel in the effective pixel region surrounded by the redundant edge pixel is determined as the target pixel. When the mosaic processing is completed, the redundant edge pixels may be clipped or other processing performed.

The above image processing method may be performed by the processor 130 in the camera as shown in fig. 1. Also, referring back to fig. 1, the camera may further include a non-transitory computer readable storage medium 140, and the non-transitory computer readable storage medium 140 may store instructions, a portion of which, when executed by the processor 130, may cause the processor 130 to perform the aforementioned image processing method.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. An image processing method, comprising:

when a target pixel is located in the image, locating a detection window centered on the target pixel, wherein the detection window has a larger pixel array specification than the color filter array;

2. The image processing method according to claim 1, wherein determining one of the first primary color, the second primary color, and the third primary color as a priority interpolation color according to the numbers of subset pixels respectively included in the first pixel subset, the second pixel subset, and the third pixel subset, comprises:

3. The image processing method of claim 1, wherein locating a target pixel in the image comprises:

any pixel in an effective pixel area surrounded by the redundant edge pixels of the image is determined as a target pixel.

4. The image processing method according to claim 1,

the peak wavelength of the first primary color is in the range of 590 nm-640 nm;

the peak wavelength of the second primary color is within the range of 500 nm-560 nm;

the peak wavelength of the third primary color is in the range of 430nm to 480 nm.

5. The image processing method of claim 4, wherein the first secondary color is obtained by mixing a first primary color and a third primary color, the second secondary color is obtained by mixing the first primary color and a second primary color, and the third secondary color is obtained by mixing the second primary color and a third primary color.

6. The image processing method according to claim 5,

the spectral power distribution of the first secondary color has a continuous wave trough section within the wavelength range of 480 nm-590 nm and continuous wave peak sections positioned at two sides of the continuous wave trough section;

the spectral power distribution of the second secondary color has a continuous valley region in a wavelength region of less than 500nm and a continuous peak region in a wavelength region of 500nm or more;

the spectral power of the third secondary color has a peak in a wavelength band of 430nm to 560nm and is attenuated from both sides of the peak.

7. The image processing method of claim 6, wherein two first color filters are diagonally arranged in the color filter array, and the second color filter and the third color filter are diagonally arranged in the color filter array in mirror image with the first color filters.

8. The method of claim 7, wherein two first color filters are arranged diagonally in the color filter array at the upper left and lower right corners, respectively, a second color filter array is arranged at the upper right corner of the color filter array, and a third color filter is arranged at the lower left corner of the color filter array.

9. The image processing method of claim 1, wherein the detection window has a pixel array specification of 3 x 3.

10. A camera, comprising: an imaging module, a processor for performing the image processing method of any one of claims 1 to 9, and the color filter element.