CN115100035A - Image processing method and device with pixel-level noise correction combined with demosaicing - Google Patents

Abstract

The invention relates to the technical field of optical engineering, in particular to a pixel-level noise correction and demosaicing combined image processing method and a device, wherein the pixel of a black-and-white channel is independently read and subjected to noise correction, an estimated image is generated by respectively using a demosaicing method based on a guide image and linear calculation on the black-and-white channel and a color channel in the image based on the corrected image, an optimized image is obtained by minimum cost function conversion, and finally the optimized image is subjected to iterative calculation to obtain an expected image; the requirements for processing pixel-level noise characteristics and accurately calculating images in a color Complementary Metal Oxide Semiconductor (CMOS) camera can be met, and images of multiple color channels in the color CMOS camera can be accurately acquired.

Description

Image processing method and device with pixel-level noise correction combined with demosaicing

Technical Field

The invention relates to the technical field of optical engineering, in particular to an image processing method and device for pixel-level noise correction and demosaicing.

Background

A Complementary Metal Oxide Semiconductor (CMOS) camera is a type of image detector commonly used in microscopic imaging, especially in the field of bioluminescence imaging. The camera is modulated by the color filter array surface, so that different pixels are differently responded by different color lights, and the single detector can capture intensity and spectrum information simultaneously, so that multicolor microscopic imaging is simple and convenient.

In color CMOS camera image processing, it is often necessary to use a demosaicing calculation method to obtain a multi-color channel image, which can then be used for multi-color image display or other needs. Kiku et al propose a Residual Interpolation method (RI) by interpolating Residual errors between estimated images and actual images of the same Color channel, which is a mosaic removing method based on a guide Image, and a mosaic removing method based on a guide Image based on Residual errors and high-frequency replacement is proposed on the basis of the patent of CN 112104847. However, in the field of bioluminescence imaging, the fluorescence signal can be as low as tens of photons, and the signal is extremely weak, approaching the electronic noise level of a camera. Limited by the CMOS single-pixel reading characteristics, the difference between pixels in the color CMOS camera is large, namely the noise at the pixel level is large; in this case, the problem of camera noise non-uniformity (i.e., pixel-level noise) due to the CMOS structure cannot be ignored, and thus, a targeted process is required. Omar A.Elgendy et al used noise correction in conjunction with demosaicing to process images in Low-Light demosaicing and demosaicing for Small Pixels Using LearnedFrequency Selection, but did not process for pixel level noise.

Some CMOS researchers developed the denoising process of pixel-level noise. The sCMOS noise-correction algorithm for the micro-vision images provides a method for correcting pixel-level noise of a black-and-white CMOS camera, and images in the black-and-white CMOS detector can be accurately acquired; CN108028895A proposes a calibration method for defective image sensor elements, which performs noise correction for extremely abnormal pixels in a camera, but the extremely abnormal pixels are only a small part, not all, of the influence of pixel-level noise; in CN111044498 patent and "Two-color super-resolution localization imaging of emission and color", based on a color CMOS camera, a single-molecule model fitting calculation process is performed on a single-molecule imaging signal (a single-point imaging signal in an image) in the color camera, and a final objective is to obtain information such as signal intensity, spatial position, etc. of each molecule, instead of obtaining an accurate multi-color channel image.

In summary, how to acquire accurate multi-color channel images by a color CMOS camera remains a problem to be solved.

Disclosure of Invention

The invention provides a pixel-level noise correction and demosaicing combined image processing method and device, which are used for overcoming at least one technical problem in the prior art.

According to a first aspect of embodiments of the present invention, there is provided an image processing method for pixel-level noise correction combined demosaicing, the method including: performing Fixed Pattern Noise (FPN) correction processing on an original image acquired by a color Complementary Metal Oxide Semiconductor (CMOS) camera based on a preset noise map corresponding to the CMOS camera to generate a first corrected image; performing white pixel readout noise correction processing on all black-and-white channel pixels in the original image based on the first correction image to generate a second correction image; generating a black and white prospective image from the second corrected image using a demosaicing method; generating a color expected image from the first corrected image using a guide image based demosaicing method, wherein the guide image is a black and white expected image.

Optionally, before the step of performing Fixed Pattern Noise (FPN) correction on the original image acquired by the color CMOS camera based on the preset noise map corresponding to the color CMOS camera to generate a first corrected image, the method further includes: and acquiring a preset noise map of the color CMOS camera.

Optionally, the step of performing Fixed Pattern Noise (FPN) correction processing on an original image acquired by a color Complementary Metal Oxide Semiconductor (CMOS) camera based on a preset noise map corresponding to the color CMOS camera to generate a first corrected image includes: based on a preset noise map corresponding to the color CMOS camera, FPN correction processing is carried out on the gray value of each pixel in a single-frame image of an original image acquired by the color CMOS camera, and a first corrected image is generated.

Optionally, the step of performing white pixel readout noise correction processing on all black-and-white channel pixels in the original image based on the first corrected image to generate a second corrected image includes: based on the first correction image, extracting all black-white channel pixels in the original image to generate and form a sampling rate reduction

A multiplied gray scale image; a second corrected image is generated by performing white pixel readout noise correction processing on the black-and-white channel pixels.

Optionally, the step of generating a black-and-white expected image from the second corrected image by using a demosaicing method includes: generating a first estimation image by using a demosaicing method and linear calculation for black and white channel pixels in the second correction image; performing minimum cost function conversion processing on the first estimation image to generate a first optimized image; and performing iterative calculation on the first optimized image to generate a black and white expected image.

Optionally, the step of generating a color expected image according to the first corrected image by using a demosaicing method based on a guide image, where the guide image is a black-and-white expected image, includes: generating a second estimated image from the first corrected image using a guide image based demosaicing method and linear computation; performing minimum cost function conversion processing on the second estimation image to generate a second optimized image; and performing iterative calculation on the second optimized image to generate a color expected image.

According to another aspect of the embodiments of the present invention, there is provided an image processing apparatus for pixel-level noise correction combined demosaicing, the apparatus including:

the device comprises a first correction module, a second correction module and a third correction module, wherein the first correction module is used for performing Fixed Pattern Noise (FPN) correction processing on an original image acquired by a color Complementary Metal Oxide Semiconductor (CMOS) camera based on a preset noise map corresponding to the color CMOS camera to generate a first corrected image;

a second correction module, configured to perform white pixel readout noise correction processing on all black-and-white channel pixels in the original image based on the first correction image, and generate a second correction image;

a preliminary generation module for generating a black and white prospective image from the second corrected image using a demosaicing method;

and a final generation module, configured to generate a color expected image according to the first corrected image by using a demosaicing method based on a guide image, where the guide image is a black-and-white expected image.

Optionally, the first correction module specifically includes a first correction submodule and a second correction submodule;

the first correction submodule is used for extracting all black-and-white channel pixels in the original image based on the first correction image to generate and form a sampling rate reduction

A multiple gray scale image;

the second correction submodule is configured to perform white pixel readout noise correction processing on the black-and-white channel pixels to generate a second correction image.

Optionally, the preliminary generation module specifically includes a first preliminary generation submodule, a second preliminary generation submodule, and a third preliminary generation submodule;

the first preliminary generation submodule is configured to generate a first estimated image using a demosaicing method and linear computation on the second corrected image;

the second preliminary generation submodule is used for carrying out minimum cost function conversion processing on the first estimation image to generate a first optimized image;

and the third preliminary generation submodule is used for carrying out iterative calculation on the first optimized image to generate a black and white expected image.

Optionally, the final generation module specifically includes a first final generation submodule, a second final generation submodule, and a third final generation submodule;

the first final generation submodule is used for generating a second estimation image according to the first correction image by using a demosaicing method based on a guide image and linear calculation;

the second final generation submodule is used for performing minimum cost function conversion processing on the second estimation image to generate a second optimized image;

and the third final generation submodule is used for performing iterative computation on the second optimized image to generate a color expected image.

The innovation points of the embodiment of the invention comprise:

1. according to the invention, for the pixel difference in the color CMOS camera, the black-white expected image and the color expected image are generated by demosaicing the black-white channel pixel and the color channel pixel respectively, so that the image acquisition accuracy of the color CMOS camera can be improved. Is one of the innovative points of the embodiment of the invention.

2. The method and the device can perform FPN correction processing on each pixel point according to the noise characteristics of each pixel point in the image based on the corresponding preset noise map in the color CMOS camera, so that a more accurate color CMOS camera image can be obtained by performing noise correction on each pixel point in the image of the color CMOS camera. Is one of the innovative points of the embodiment of the invention.

3. The method comprises the steps of performing spectrum measurement on noise of the color CMOS camera to evaluate actual image noise, performing preliminary estimation by a demosaicing method based on a guide image, generating an estimated image, performing minimum cost function conversion processing on the estimated image to generate an optimized image, performing multiple iterative optimization calculations on the optimized image, and finally accurately acquiring a multi-color channel image in the color CMOS camera, so that the requirements for pixel-level noise feature processing and image accurate calculation in the color CMOS camera can be met. Is one of the innovative points of the embodiment of the invention.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

FIG. 1 is a schematic processing flow diagram of an image processing method of pixel-level noise correction combined with demosaicing according to the present invention;

FIG. 2 is a schematic processing flow diagram of another pixel-level noise correction combined demosaicing image processing method according to the present invention;

FIG. 3 is a flowchart illustrating a detailed process of step 201 in the present invention;

FIG. 4 is a diagram of a color wavefront profile for a color CMOS camera according to the present invention;

FIG. 5 is a flow chart of a method for demosaicing and pixel noise corrected image restoration;

FIG. 6 is a schematic structural diagram of an image processing apparatus for pixel-level noise correction combined with demosaicing according to the present invention;

FIG. 7 is a schematic structural diagram of a first calibration module according to the present invention;

FIG. 8 is a schematic structural diagram of a preliminary generation module according to the present invention;

fig. 9 is a schematic structural diagram of a final generating module according to the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without inventive effort based on the embodiments of the present invention, are within the scope of the present invention.

It is to be noted that the terms "comprises" and "comprising" and any variations thereof in the embodiments and drawings of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.

Example 1

The embodiment of the present invention provides a first image processing method based on pixel-level noise correction and demosaicing, and referring to fig. 1, fig. 1 is a schematic processing flow diagram of an image processing method based on pixel-level noise correction and demosaicing according to the present invention. As shown in fig. 1, the image processing method of pixel-level noise correction combined demosaicing includes the following steps:

step 101, based on a preset noise map corresponding to a color CMOS camera, performing Fixed Pattern Noise (FPN) correction processing on an original image acquired by the color CMOS camera to generate a first corrected image.

It should be noted that the preset noise map corresponding to the color CMOS camera can be generated by measuring the relative photon response value of each pixel in the color CMOS camera and reading the noise value and the relative offset value.

In this step, the gray value of each pixel in the single-frame picture of the original image may be corrected through a photon response value and a relative offset value in a preset noise map obtained in advance, so as to implement Fixed Pattern Noise (FPN) correction of the original image, and generate a first corrected image after FPN correction.

It can be understood that the first corrected image is obtained by performing noise correction on each pixel point in the original image of the color CMOS camera, and compared with the prior art in which the pixel points are not processed in a distinguishing manner, the noise correction of each pixel point in the present invention can make the generated color CMOS image more accurate.

And 102, performing white pixel reading noise correction processing on all black-and-white channel pixels in the original image based on the first correction image to generate a second correction image.

In this step, based on the first corrected image obtained in step 101, white pixel read noise correction processing is performed on all the black-and-white channel pixels of the original image, and a second corrected image after the white pixel read noise correction processing is generated.

It should be noted that the color CMOS camera is a color CMOS camera arranged in a W-R-G-B-NIR array. The W-R-G-B-NIR array camera comprises a black-white channel (W) and a color channel. Since there are many black and white channel (W) pixels in the color filter array surface of the color CMOS camera, all the black and white channel (W) pixels can be extracted and then directly corrected by using a scientific research-grade CMOS noise correction method (NCS).

Step 103, generating a black and white prospective image from said second corrected image using a demosaicing method.

In this step, the second corrected image obtained in step 102 may be processed based on a preset demosaicing method to generate a black-and-white expected image.

A color intended image is generated from the first corrected image using a guide image based demosaicing method, step 104.

Wherein the guide image is a black and white prospective image.

In this step, the first corrected image obtained in step 101 may be processed based on a demosaicing method of a preset guide image to generate a color expected image. The guide image in this step may be the black and white expected image generated in step 103.

It should be noted that the demosaicing method may be implemented by using various mathematical methods, and in a specific implementation process, the demosaicing method may use a residual interpolation method. The demosaicing method based on the guide image can refer to the related descriptions in steps 204 and 205 in embodiment 2 of the present invention.

In the invention, considering that the pixel difference in the color CMOS camera is large, in order to further improve the image accuracy of the color CMOS camera, black and white channel pixels and color channel pixels can be distinguished, and pixel-level noise correction and demosaicing processing based on a guide image are respectively carried out so as to generate a more accurate expected image.

In a specific implementation, the black-and-white prospective image generated in step 103 and the color prospective image generated in step 104 can be further processed according to the requirements of the actual application. For example, the black-and-white expected image and the color expected image can be combined into an accurate color image based on a full-color (RGB) image, and for example, in the field of fluorescence microscopy, since different fluorescent dye types are to be distinguished during imaging, the fluorescent dye types of an imaged object can be analyzed based on the intensity difference of signal values in different color channel images of the same pixel in the black-and-white expected image and the color expected image. The invention is not limited with respect to the subsequent processing of black and white prospective images and color prospective images.

It should be noted that Single-shot multi-color fluorescence micro vision a color measurement camera proposes to process an image in a color CMOS camera by using a demosaicing method, but the method processes the image on the premise that all pixels in the image under the same color channel are considered to be the same pixel, and pixel-level noise is generated in the color CMOS camera due to the pixel difference problem, so the processing effect of the method is not ideal.

Therefore, the embodiment of the invention can generate the black-white expected image and the color expected image by demosaicing the black-white channel pixel and the color channel pixel respectively according to the pixel difference in the color CMOS camera, and can improve the accuracy of the color CMOS camera in acquiring the images; in addition, the embodiment of the invention can also perform FPN correction processing on each pixel point according to the noise characteristic of each pixel point in the image based on the corresponding preset noise map in the color CMOS camera, so as to obtain a more accurate color CMOS camera image by performing noise correction on each pixel point in the image of the color CMOS camera.

Example 2

An embodiment of the present invention provides another image processing method based on pixel-level noise correction and demosaicing, and referring to fig. 2, fig. 2 is a schematic processing flow diagram of another image processing method based on pixel-level noise correction and demosaicing according to the present invention. As shown in fig. 2, the image processing method of pixel-level noise correction combined demosaicing includes the following steps:

step 201, measuring a noise map of a preset color CMOS camera to obtain the noise map of the color CMOS camera.

In the step, a noise map of the color CMOS camera is generated by measuring the relative photon response value of each pixel in the color CMOS camera and reading the noise value and the relative deviation value, so that a standard basis for correction is provided for the subsequent noise correction of each pixel point.

It should be noted that the noise map is composed of a photon response map, a read noise map and a relative offset map, wherein the photon response map is composed of relative photon response values of each pixel in the color CMOS camera, the read noise map is composed of read noise values of each pixel in the color CMOS camera, and the relative offset map is composed of relative offset values of each pixel in the color CMOS camera.

Optionally, referring to fig. 3, fig. 3 is a flowchart of a specific processing in step 201 in the present invention, and as shown in fig. 3, step 201 may specifically include:

and a substep 11 of obtaining an offset value and a readout noise value for each pixel based on the known gray scale value output from the color CMOS camera.

In this step, when the color CMOS camera is not added with the signal light source, the gradation value of the color CMOS camera is determined only by the offset value and the readout noise value, and therefore, the offset value and the readout noise value of each pixel can be found by the known gradation value.

In a specific implementation, the average and root mean square values of N consecutive frames of non-incident light images may be used to represent the offset value and readout noise value for each pixel.

Specifically, the offset value of each pixel can be calculated by formula (1), and the readout noise value of each pixel can be calculated by formula (2):

wherein, X _i,j,dark Representing the gray value of the pixel i in the camera in the j-th dark frame; off _i An offset value representing pixel i; sigma _i,read Represents the readout noise value of pixel i; n is the number of images, and when the exposure time is 1s or less, N is 5000, and when the exposure time is more than 1s, N is 1000.

And a substep 12 of providing a stable light signal by adjusting the illumination intensity of a preset uniform illumination system and eliminating random noise of individual pixels by using the mean gray value to obtain a relative photon response value of each pixel.

In this step, the relative photon response value of each pixel is measured using a homogeneous illumination system. Assuming that the photon signals of all pixels in a frame are the same, the average gray value of successive images is used

To eliminate random noise. The relative photon response value of each pixel is the ratio between the signal value of a single pixel and the average of all pixels.

Specifically, the relative photon response value of each pixel can be calculated by formula (3):

wherein NP is the number of all pixels; rp _i Is the relative photon response value of pixel i; off _i An offset value for pixel i;

is the average gray value of pixel i in the NP frame image.

It should be noted that for the purpose of calculation, inEquation (3) uses a linear fit of the relative photon responses of several sets of images taken at different light signal intensity levels, per pixel

As the dependent variable, the amount of the reaction,

as an independent variable.

In a specific implementation process, 6 groups of original images can be selected to be shot, each group comprises 1000 frames, the illumination intensity of the uniform illumination system is set to be 10% -85% of the whole signal range, and the signal range is the signal range which can be measured by the camera.

It should be noted that the measurement of the noise map needs to correspond to the experimental conditions, including exposure time, fluorescence wavelength, camera settings, etc., so that the illumination intensity can be adjusted to generate a photon response map matching the experimental conditions.

Step 202, based on a preset noise map corresponding to the color CMOS camera, performing FPN correction processing on a gray value of each pixel in a single frame image of the original image acquired by the color CMOS camera to generate a first corrected image.

In this step, the gray value of each pixel in a single-frame picture of an original image acquired by a color CMOS camera is subjected to FPN correction processing through a photon response map and a relative shift map obtained by measurement, and a first corrected image of the original image after the FPN correction processing is generated.

Specifically, the gradation value of each pixel can be corrected by formula (4):

X _i,dF ＝(X _i,raw -Off _i )/rp _i (4)

wherein, X _i,dF The corrected gray value for pixel i; x _i,raw The gray value before the correction is the pixel i; rp _i Is the relative photon response value, Off, of the pixel i _i Is the offset value for pixel i. Rp when said photon response map is absent _i Has a value of 1.

It can be understood that the first corrected image is obtained by performing noise correction on each pixel point in the original image of the color CMOS camera, and compared with the prior art in which no distinction processing is performed on the pixel points, the color CMOS image generated by the present invention is more accurate.

Step 203, based on the first corrected image, extracting all black and white channel pixels in the original image, and generating and forming a sampling rate reduction

Multiple gray scale image.

In this step, since there are more black and white channel (W) pixels and fewer color channel pixels in the color filter array surface of the CMOS camera, all the black and white channel (W) pixels can be extracted first and then changed into a single pixel with a reduced sampling rate

Multiple gray scale image.

It should be noted that the color channels specifically include: red channel (R), green channel (G), blue channel (B), and near infrared channel (NIR). As shown in fig. 4, fig. 4 is a color wavefront distribution diagram of the color CMOS camera of the present invention, the size of the local structure is 4 × 4, the black-and-white channels and the color channels are uniformly distributed, the arrangement of the first row of channels is specifically R-W-G-W, the arrangement of the second row of channels is specifically W-NIR-W-NIR, the arrangement of the third row of channels is specifically G-W-B-W, and the arrangement of the fourth row of channels is specifically W-NIR-W-NIR; the color responsivity of each color channel is different under different wavelengths.

Step 204, generating a second corrected image by performing a white pixel readout noise correction process on the black-and-white channel pixels.

In this step, first, the dip can be corrected using the NCS noise correction method

The doubled grayscale image is subjected to white pixel readout noise correction processing, and then the pixel values of the corrected grayscale image are used to replace the corresponding black-and-white channel pixel values in the first corrected image.

Step 205, generating a first estimated image using a demosaicing method and linear calculation on the second corrected image.

In this step, first, a white interpolation image may be calculated for the second correction image using a method of calculating a guide image in a guide image-based demosaicing method, and then the white interpolation image and the first correction image may be linearly calculated to generate a first estimation image.

Specifically, the first estimated image can be obtained by formula (5):

X _e1 ＝∈X _it +(1-∈)X _dF (5)

wherein, X _e1 Is the first estimated image; x _it Interpolating the white color image; x _dF Is the first corrected image; e is a coefficient, and can be set to 0.1.

Specifically, as shown in fig. 5, fig. 5 is a flowchart of a demosaicing and pixel noise correction image restoration method. The Optical Transfer Function (OTF) denoising method is characterized in that the whole image is calculated to be influenced by noise based on an optical transfer function mask, and simultaneously, a denoised image after high-frequency filtering of the OTF mask is calculated. The maximum likelihood estimation refers to estimating the probability of the actual signal value of the original image based on the pixel value of the experimental image and an image noise model, and the maximum likelihood estimation can simultaneously consider the influence of the experimental image and pixel-level noise.

It should be noted that there are many specific implementation methods for the demosaicing method based on the guide image, and usually, the complete guide image is calculated by the steps of interpolation and the like based on the original filtered front image, and then the other color channel images are calculated based on the guide image.

And step 206, performing minimum cost function conversion processing on the first estimation image to generate a first optimized image.

Specifically, the first estimated image is used as an initial value of the estimated image, the value of the estimated image is continuously changed in the calculation process, the cost function is calculated, and the corresponding estimated image when the cost function is minimized is used as the first optimized image. The cost function is shown in equation (6):

f＝LLS _C +ασ _NP +βdm _NP-C (6)

where α is a coefficient, which may be set to 1; β is a coefficient, which can be set to 0.01; LLS _C A simplified negative log-likelihood function; sigma _NP Is a noise contribution value; dm _NP-C Contributing values to demosaicing; f is the cost function.

It should be noted that LLS _C The calculation of (2) requires the numerical information of the processed image, and when the black and white pixel channel is calculated, the calculation can be only carried out aiming at all white pixels; when calculating color channels, the calculation can be performed only for all color pixels, wherein the processed image is the estimated image.

Specifically, LLS can be obtained by the formula (7) _C ：

Wherein LLS _C For the simplified negative log-likelihood function, C represents all pixels in the currently computed color channel (W in this step) in the color-filter-front mode, σ _i,read ² The representative pixel i reads out the noise electronic value.

For the value of said first correction image pixel i,

is the value of the estimated image pixel i.

σ _N Specifically, the noise part energy filtered by the OTF wavefront can be obtained by equation (8):

wherein NP represents all pixels; ot (a)f represents the image after being filtered by the high-pass raised cosine filter.

Is the value of the estimated image pixel i.

Specifically, the high-pass raised cosine filter is shown in formula (9):

wherein, M (k) _x ,k _y ) Representing the high-pass raised cosine filter. k is a radical of _r Is a plane wave vector and is a wave vector of the plane,

k _x and k _y Respectively representing wave vectors of the image in the x and y directions; β is 1, T ═ λ/5.6 NA; and lambda and NA are the wavelength of the imaging object and the numerical aperture of the imaging microscope objective.

dm _NP-C Representing the difference between the estimated image and the interpolated image, dm can be obtained by equation (10) _NP-C ：

Where NP-C represents the pixels of all channels except the currently computed color channel.

For the value of the estimated image pixel i,

is the value of the interpolated image pixel i.

And step 207, performing iterative computation on the first optimized image to generate a black-and-white expected image.

Specifically, the first optimized image may be used as an estimated image, and the first optimized image may be subjected to loop iteration minimization cost function conversion processing to generate a black-and-white expected image.

In particular implementations, the number of loop iterations may be 10-15.

A second estimated image is generated from the first corrected image using a guide image based demosaicing method and linear computation, step 208.

Wherein the guide image is a black and white prospective image.

In this step, first, a color interpolation image may be calculated for the first correction image by using a method of calculating the guide image in the demosaic method based on the guide image, and then the color interpolation image and the first correction image may be linearly calculated to generate the second estimation image.

Specifically, the specific processing procedure in this step may refer to step 205, which is not described herein again.

Step 209, performing a minimum cost function transformation process on the second estimation image to generate a second optimized image.

Specifically, the specific processing procedure in this step may refer to step 206, which is not described herein again.

And 210, performing iterative computation on the second optimized image to generate a color expected image.

In this step, the second optimized image may be used as a second estimated image, and the second optimized image may be subjected to a loop iteration minimization cost function conversion process to generate a color expected image.

In particular implementations, the number of loop iterations may be 10-15.

It should be noted that the black-and-white expected image and the color expected image can be further processed according to the requirements of the actual application.

It should be further noted that, step 203 specifies the distribution of the color channels, and that the conducting of the demosaicing process based on the guide map for the color channel pixels in steps 208 to 210 is an operation process for a single one of the color channels. Specifically, the other three color channels can be processed with reference to steps 208 to 210 according to the actual application requirement.

Therefore, the embodiment of the invention can evaluate the actual image noise by performing spectrum measurement on the noise of the color CMOS camera, preliminarily estimate the image by a demosaicing method based on a guide image, generate an estimated image, perform minimum cost function conversion processing on the estimated image to generate an optimized image, perform multiple iterative optimization calculation on the optimized image, and finally accurately acquire the multi-color channel image in the color CMOS camera, thereby meeting the requirements for pixel-level noise feature processing and image accurate calculation in the color CMOS camera.

Example 3

An embodiment of the present invention provides a first image processing apparatus for pixel-level noise correction and demosaicing, and referring to fig. 6, fig. 6 is a schematic structural diagram of the apparatus of the present invention. As shown in fig. 6, the apparatus 90 includes:

a first correction module 901, configured to perform Fixed Pattern Noise (FPN) correction processing on an original image acquired by a color Complementary Metal Oxide Semiconductor (CMOS) camera based on a preset noise map corresponding to the CMOS camera, so as to generate a first corrected image;

a second correction module 902, configured to perform white pixel readout noise correction processing on all black-and-white channel pixels in the original image based on the first correction image, and generate a second correction image;

a preliminary generation module 903 for generating a black and white prospective image from the second corrected image using a demosaicing method;

a final generating module 904, configured to generate a color expected image according to the first corrected image by using a demosaicing method based on a guide image, where the guide image is a black-and-white expected image.

Optionally, as shown in fig. 7, the first correction module 901 specifically includes a first correction submodule 911 and a second correction submodule 912;

the first correction submodule 911 is configured to extract all black and white channel pixels in the original image based on the first corrected image, and generate a sampling rate reduction

A multiplied gray scale image;

the second correction sub-module 912 is configured to generate a second correction image by performing a white pixel readout noise correction process on the black-and-white channel pixels.

Optionally, as shown in fig. 8, the preliminary generation module 903 specifically includes a first preliminary generation sub-module 931, a second preliminary generation sub-module 932, and a third preliminary generation sub-module 933;

the first preliminary generation sub-module 931 is configured to generate a first estimation image using a demosaicing method and linear computation on the second correction image;

the second preliminary generation submodule 932 is configured to perform minimum cost function conversion processing on the first estimation image to generate a first optimized image;

the third preliminary generation submodule 933 is configured to perform iterative computation on the first optimized image to generate a black-and-white expected image.

Optionally, as shown in fig. 9, the final generating module 904 specifically includes a first final generating sub-module 941, a second final generating sub-module 942, and a third final generating sub-module 943;

the first final generation sub-module 941 is configured to generate a second estimated image from the first corrected image using a guided image-based demosaicing method and linear calculation;

the second final generation sub-module 942 is configured to perform a minimum cost function transformation process on the second estimated image to generate a second optimized image;

the third final generation submodule 943 is configured to perform iterative calculations on the second optimized image to generate a color expected image.

Therefore, the pixel-level noise correction and demosaicing combined image processing device provided by the embodiment of the invention can generate a black-and-white expected image and a color expected image by demosaicing the black-and-white channel pixels and the color channel pixels respectively according to the pixel difference in the color CMOS camera, and can improve the image acquisition accuracy of the color CMOS camera; in addition, the embodiment of the invention can also perform FPN correction processing on each pixel point according to the noise characteristic of each pixel point in the image based on the corresponding preset noise map in the color CMOS camera, so as to obtain a more accurate color CMOS camera image by performing noise correction on each pixel point in the image of the color CMOS camera.

Those of ordinary skill in the art will understand that: the figures are only schematic representations of one embodiment, and the blocks or flows in the figures are not necessarily required to practice the present invention.

Those of ordinary skill in the art will understand that: modules in the devices in the embodiments may be distributed in the devices in the embodiments according to the description of the embodiments, or may be located in one or more devices different from the embodiments with corresponding changes. The modules of the above embodiments may be combined into one module, or may be further split into a plurality of sub-modules.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. A method for pixel-level noise correction in conjunction with demosaicing, the method comprising:

performing Fixed Pattern Noise (FPN) correction processing on an original image acquired by a color Complementary Metal Oxide Semiconductor (CMOS) camera based on a preset noise map corresponding to the CMOS camera to generate a first corrected image;

performing white pixel readout noise correction processing on all black-and-white channel pixels in the original image based on the first correction image to generate a second correction image;

generating a black and white prospective image from the second corrected image using a demosaicing method;

generating a color expected image from the first corrected image using a guide image based demosaicing method, wherein the guide image is a black and white expected image.

2. The method of claim 1, wherein before the step of performing Fixed Pattern Noise (FPN) correction on the raw image acquired by the color CMOS camera based on the corresponding preset noise map of the color CMOS camera to generate the first corrected image, the method further comprises:

and acquiring a preset noise map of the color CMOS camera.

3. The method of claim 1, wherein the step of performing a Fixed Pattern Noise (FPN) correction process on an original image acquired by a color Complementary Metal Oxide Semiconductor (CMOS) camera based on a corresponding preset noise map of the color CMOS camera to generate a first corrected image comprises:

based on a preset noise map corresponding to the color CMOS camera, FPN correction processing is carried out on the gray value of each pixel in a single-frame image of an original image acquired by the color CMOS camera, and a first corrected image is generated.

4. The method according to claim 1, wherein the step of performing white pixel readout noise correction processing on all black-and-white channel pixels in the original image based on the first corrected image to generate a second corrected image comprises:

based on the first correction image, extracting all black-white channel pixels in the original image to generate and form a sampling rate reduction

A multiplied gray scale image;

a second corrected image is generated by performing white pixel readout noise correction processing on the black-and-white channel pixels.

5. The method according to claim 1, wherein said step of generating a black-and-white intended image from said second corrected image, using a demosaicing method, comprises:

generating a first estimation image by using a demosaicing method and linear calculation for black and white channel pixels in the second correction image;

performing minimum cost function conversion processing on the first estimation image to generate a first optimized image;

and performing iterative calculation on the first optimized image to generate a black and white expected image.

6. The method according to claim 1, wherein the step of generating a color expected image from the first corrected image using a guide image based demosaicing method, wherein the guide image is a black and white expected image, comprises:

generating a second estimated image from the first corrected image using a guide image based demosaicing method and linear computation;

performing minimum cost function conversion processing on the second estimation image to generate a second optimized image;

and performing iterative calculation on the second optimized image to generate a color expected image.

7. An image processing apparatus that performs pixel-level noise correction in conjunction with demosaicing, the apparatus comprising:

the device comprises a first correction module, a second correction module and a third correction module, wherein the first correction module is used for performing Fixed Pattern Noise (FPN) correction processing on an original image acquired by a color Complementary Metal Oxide Semiconductor (CMOS) camera based on a preset noise map corresponding to the color CMOS camera to generate a first corrected image;

a second correction module, configured to perform white pixel readout noise correction processing on all black-and-white channel pixels in the original image based on the first correction image, and generate a second correction image;

a preliminary generation module for generating a black and white prospective image from the second corrected image using a demosaicing method;

and a final generation module, configured to generate a color expected image according to the first corrected image by using a demosaicing method based on a guide image, where the guide image is a black-and-white expected image.

8. The apparatus of claim 7,

the first correction module specifically comprises a first correction submodule and a second correction submodule;

the first correction submodule is used for extracting all black-and-white channel pixels in the original image based on the first correction image to generate and form a sampling rate reduction

A multiplied gray scale image;

the second correction submodule is configured to perform white pixel readout noise correction processing on the black-and-white channel pixels to generate a second correction image.

9. The apparatus of claim 7,

the preliminary generation module specifically comprises a first preliminary generation submodule, a second preliminary generation submodule and a third preliminary generation submodule;

the first preliminary generation submodule is used for generating a first estimation image by using a demosaicing method and linear calculation on black-and-white channel pixels in the second correction image;

the second preliminary generation submodule is used for carrying out minimum cost function conversion processing on the first estimation image to generate a first optimized image;

and the third preliminary generation submodule is used for carrying out iterative calculation on the first optimized image to generate a black and white expected image.

10. The apparatus of claim 7,

the final generation module specifically comprises a first final generation submodule, a second final generation submodule and a third final generation submodule;

the first final generation submodule is used for generating a second estimation image according to the first correction image by using a demosaicing method based on a guide image and linear calculation;

the second final generation submodule is used for performing minimum cost function conversion processing on the second estimation image to generate a second optimized image;

and the third final generation submodule is used for carrying out iterative calculation on the second optimized image to generate a color expected image.

Images