CN116228591A - Method and device for eliminating crosstalk between pixels of image sensor - Google Patents
Method and device for eliminating crosstalk between pixels of image sensor Download PDFInfo
- Publication number
- CN116228591A CN116228591A CN202310296645.3A CN202310296645A CN116228591A CN 116228591 A CN116228591 A CN 116228591A CN 202310296645 A CN202310296645 A CN 202310296645A CN 116228591 A CN116228591 A CN 116228591A
- Authority
- CN
- China
- Prior art keywords
- crosstalk
- convolution
- mathematical model
- image function
- function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000013178 mathematical model Methods 0.000 claims abstract description 67
- 238000004364 calculation method Methods 0.000 claims abstract description 25
- 238000001914 filtration Methods 0.000 claims abstract description 13
- 230000008569 process Effects 0.000 claims abstract description 13
- 230000009466 transformation Effects 0.000 claims description 12
- 239000000654 additive Substances 0.000 claims description 6
- 230000000996 additive effect Effects 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 6
- 238000003384 imaging method Methods 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 3
- 230000000694 effects Effects 0.000 claims description 2
- 238000002474 experimental method Methods 0.000 claims description 2
- 230000006870 function Effects 0.000 description 98
- 238000010586 diagram Methods 0.000 description 10
- 238000002955 isolation Methods 0.000 description 8
- 230000008030 elimination Effects 0.000 description 4
- 238000003379 elimination reaction Methods 0.000 description 4
- 238000012938 design process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 2
- 238000009792 diffusion process Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 229910052710 silicon Inorganic materials 0.000 description 2
- 239000010703 silicon Substances 0.000 description 2
- 239000000758 substrate Substances 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 239000012212 insulator Substances 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/15—Correlation function computation including computation of convolution operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20048—Transform domain processing
- G06T2207/20056—Discrete and fast Fourier transform, [DFT, FFT]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Mathematical Physics (AREA)
- Computational Mathematics (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Algebra (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Facsimile Image Signal Circuits (AREA)
Abstract
The invention relates to a method and a device for eliminating crosstalk between pixels of an image sensor. The method comprises the following steps: establishing a crosstalk convolution mathematical model between an actual image function and an ideal image function of an image sensor pixel, acquiring a regular filter coefficient based on a convolution kernel in advance, and further acquiring and storing a crosstalk coefficient; acquiring an actual image function acquired by an image sensor; and performing convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficient, obtaining and outputting an ideal image function with crosstalk between pixels eliminated. The invention converts the crosstalk problem between pixels into the convolution problem by establishing the crosstalk convolution mathematical model, and can eliminate the crosstalk between pixels by using the same convolution kernel and adopting a regular filtering mode to restore an ideal image under the condition of not changing the pixel structure for the image sensor with the same process condition.
Description
Technical Field
The present invention relates to the field of ranging technologies, and in particular, to a method and an apparatus for eliminating crosstalk between pixels of an image sensor.
Background
The Time-of-Flight (ToF) technique is a method for precisely measuring the distance of a reference object, and there are direct-Time-of-Flight (dtif) ranging techniques, i.e., directly measuring the Time of Flight of light to calculate the distance of the reference object, and indirect-Time-of-Flight (iToF) ranging techniques, i.e., periodically modulating and demodulating the light intensity and then calculating the distance of the reference object using phase information.
Please refer to fig. 1, which is a schematic diagram illustrating an operation principle of the IToF image sensor. As shown in fig. 1, the IToF image sensor operates according to the following principle: controlling an emitter 12 of an image Sensor (Sensor) by a Modulation module (Modulation) 11 to actively generate emitted light 13; the emitted light 13 is emitted to the surface of the object 19 to be detected, and reflected light 14 formed after being reflected by the object 19; the signal of the reflected light 14 is sampled and acquired by a detector 15 of the image sensor; and further based on the relative phase difference of the emitted light 13 and the reflected light 14
To calculate depth information of the object 19. Wherein the emitter 12 generally generates emitted light using an active light source in the near infrared wavelength band; the detector is typically an array of pixels of an image sensor.
Please refer to fig. 2, which is a schematic diagram of a pixel structure of an iToF image sensor in the prior art, wherein a solid line with an arrow represents incident light, and a dashed curve with an arrow represents an electron motion path. As shown in fig. 2, the ToF image sensor pixel structure includes: a Photodiode (photo diode) 22 formed in the silicon substrate 21, a dielectric layer 23 covering the silicon substrate 21, and a microlens 24 formed on the dielectric layer 23. The reflected light reflected back at the pixel's corresponding reference location enters the pixel as incident light 29, which incident light 29 generates photo-generated electrons 28 within the pixel and is output in the form of a voltage signal. Ideally, the reflected light reflected by the pixel corresponding to the reference object position is collected only by the pixel, however, in practical application, due to the fact that optical and electrical complete isolation is not performed between the pixels, due to light incidence or electron diffusion, a part of photons or electrons are collected by surrounding pixels, and Crosstalk (CTK) between the pixels is generated.
With the development of iToF image sensors and consumer demand, high Resolution (Resolution) iToF image sensors have been the target pursued by numerous manufacturers and consumers. The high resolution means that the pixel size is reduced, which causes an increase in crosstalk between pixels, causes a ranging error of the iToF image sensor, and affects the ranging accuracy. Therefore, eliminating crosstalk between pixels is critical for high quality iToF image sensors.
In the prior art, a scheme for eliminating crosstalk between pixels is generally that a deep channel isolation (Deep Trench Isolation, abbreviated as DTI) structure is added between pixels in the pixel design process, so that two adjacent pixels are isolated from a physical structure. The DTI structure is generally made of an insulator, and can block diffusion of photo-generated electrons between pixels and eliminate electrical crosstalk between pixels, so that crosstalk between pixels is reduced. However, DTI structures have limited effectiveness in blocking incident light from other surrounding pixels and do not completely eliminate optical crosstalk. In addition, DTI structures need to be introduced during the pixel design process, and crosstalk cancellation cannot be performed on existing image sensors.
Therefore, how to perform crosstalk cancellation on the existing image sensor is a technical problem to be solved currently.
Disclosure of Invention
The invention aims to provide a method and a device for eliminating crosstalk between pixels of an image sensor, which can effectively eliminate the crosstalk between pixels and can eliminate the crosstalk of the existing image sensor on the basis of not designing an additional deep channel isolation structure.
In order to achieve the above object, the present invention provides a method for eliminating crosstalk between pixels of an image sensor, including: establishing a crosstalk convolution mathematical model between an actual image function and an ideal image function of an image sensor pixel, acquiring a regular filter coefficient based on a convolution kernel in advance, and further acquiring and storing a crosstalk coefficient; acquiring an actual image function acquired by an image sensor; and performing convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficient, obtaining and outputting an ideal image function with crosstalk between pixels eliminated.
In order to achieve the above object, the present invention further provides a device for eliminating crosstalk between pixels of an image sensor, including: the model building module is used for building a crosstalk convolution mathematical model between an actual image function and an ideal image function of the image sensor pixel, acquiring a regular filter coefficient in advance, further acquiring a crosstalk coefficient and storing the crosstalk coefficient; the acquisition module is used for acquiring an actual image function acquired by the image sensor; and the processing module is used for carrying out convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficient, obtaining and outputting an ideal image function with crosstalk between pixels eliminated.
To achieve the above object, the present invention also provides an electronic device including a memory, a processor, and a computer executable program stored on the memory and executable on the processor; the processor, when executing the computer executable program, implements the steps of the method for eliminating crosstalk between pixels of an image sensor according to the present invention.
According to the invention, the crosstalk problem between pixels is converted into a convolution problem by establishing a crosstalk convolution mathematical model, when crosstalk between pixels is eliminated for an image sensor with the same process condition, an ideal image is restored by acquiring a prestored regular filter coefficient and acquiring a Fourier transform function corresponding to an actual image function acquired by the image sensor, and obtaining an approximate solution of the ideal image function through 2-step calculation, wherein the crosstalk between pixels is eliminated in a regular filter mode under the condition that the pixel structure is not changed; or further optimizing an image crosstalk elimination algorithm, obtaining an approximate solution of an ideal image function through 1-step calculation by obtaining a prestored target sequence and obtaining an actual image function acquired by the image sensor, and recovering an ideal image on the basis of reducing the calculated amount of the image sensor. The invention can effectively eliminate crosstalk between pixels on the basis of not needing to design an extra deep channel isolation structure, and can eliminate the crosstalk of the existing image sensor, thereby having wider application range. Meanwhile, the invention can also be applied to the image sensor after the scheme of eliminating crosstalk between pixels by adopting the deep channel isolation structure, so as to realize further elimination of the crosstalk.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that are required to be used in the description of the embodiments will be briefly described below. It is evident that the drawings in the following description are only some embodiments of the present invention and that other drawings may be obtained from these drawings without inventive effort for a person of ordinary skill in the art.
Fig. 1 is a schematic diagram of the working principle of an iToF image sensor;
FIG. 2 is a schematic diagram of a pixel structure of an iToF image sensor in the prior art;
FIG. 3 is a schematic diagram of a method for eliminating crosstalk between pixels of an image sensor according to the present invention;
FIG. 4 is a diagram showing a relationship between an ideal signal and an output signal according to an embodiment of the present invention;
fig. 5 is a block diagram of a device for eliminating crosstalk between pixels of an image sensor according to the present invention.
Detailed Description
The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to fall within the scope of the invention.
An embodiment of the invention provides a method for eliminating crosstalk between pixels of an image sensor.
Referring to fig. 3 to fig. 4 together, fig. 3 is a schematic diagram of steps of a method for eliminating crosstalk between pixels of an image sensor according to the present invention, and fig. 4 is a schematic diagram of a relationship between an ideal signal and an output signal according to an embodiment of the present invention.
As shown in fig. 3, the method for eliminating crosstalk between pixels of an image sensor according to the present embodiment includes the following steps: s1, establishing a crosstalk convolution mathematical model between an actual image function and an ideal image function of an image sensor pixel, acquiring a regular filter coefficient in advance, and further acquiring and storing a crosstalk coefficient; s2, acquiring an actual image function acquired by an image sensor; s3, carrying out convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficient, obtaining and outputting an ideal image function with crosstalk between pixels eliminated. A detailed explanation is given below.
Regarding step S1, a crosstalk convolution mathematical model between an actual image function and an ideal image function of the image sensor pixel is established, and a regular filter coefficient is obtained in advance so as to obtain and store the crosstalk coefficient. Specifically, by establishing a crosstalk convolution mathematical model, the crosstalk problem between pixels can be converted into a convolution problem, and according to an ideal image, an actual image output by the image sensor can be obtained through a certain convolution kernel.
As shown in fig. 4, taking the iToF image sensor as an example, a description is given of a principle of crosstalk convolution mathematical model establishment. The pixel array of fig. 4 simulates a real-world pixel array using the pixel array of 3*3, with the upper part representing information for each pixel in an ideal situation and the lower part representing other pixels around the periphery affected by the information for that pixel in a real application. Part (a) in fig. 4 represents the pixel range 421 affected by the first pixel 411; part (b) of fig. 4 represents the pixel range 422 affected by the second pixel 412; part (c) of fig. 4 represents the pixel range 429 affected by the last pixel 419. During operation of the iToF image sensor, instead of only one pixel at a time, the pixels of the entire pixel array are operated simultaneously. This results in each pixel collecting not only its own signal, but also the signals that crosstalk from other pixels around it to that pixel, ultimately measuring the signal output by the pixel array.
Thus, the relationship between the ideal signal and the actual output signal for each pixel can be described by:
wherein ,Amn_output Output signals for pixels at positions of the pixel array having coordinates (m, n); b (B) ij_ideal Is an ideal signal for a pixel at the position of coordinates (i, j) in the pixel array; a, a mn The coefficient of influence for all pixels in the pixel array on the pixel at the (m, n) position of coordinates. The signal output by the final pixel can be understood as a superposition of all pixel signal components in space, so that crosstalk between pixels can be expressed in terms of convolution.
From the above analysis, it can be seen that: by establishing a crosstalk convolution mathematical model, the crosstalk problem between pixels can be converted into a convolution problem, and according to an ideal image, an actual image output by the iToF image sensor can be obtained through a certain convolution kernel.
In some embodiments, the crosstalk convolution mathematical model is expressed using the following formula:
g (x, y) =h (x, y) ×f (x, y) +n (x, y) (formula 1)
Wherein g (x, y) is the actual image function, f (x, y) is the ideal image function, h (x, y) is a convolution kernel, and n (x, y) is additive noise.
In some embodiments, the canonical filter coefficients are further obtained by: acquiring a convolution kernel through an imaging experiment of a point light source on a central pixel of a pixel array of an image sensor in advance, and acquiring a regular filter coefficient according to the convolution kernel; wherein the regular filter coefficients are the same for image sensors of the same process conditions when eliminating crosstalk between pixels.
Specifically, the convolution kernel h (x, y) is obtained by: (1) Measuring crosstalk array CTK (x, y) of a central pixel point of the pixel array on the whole image; (2) The crosstalk array CTK (x, y) is divided by the sum of all elements in the crosstalk array to obtain a convolution kernel h (x, y).
Wherein the crosstalk CTK (x, y) is divided by the sum of all elements in the crosstalk CTK (x, y) is represented by the following formula:
wherein, M and N are the number of rows and columns of crosstalk array CTK (x, y), respectively.
In some embodiments, the canonical filter coefficient is expressed using the following formula:
wherein HP is the canonical filter coefficient, H (u, v) is the Fourier transform of the convolution kernel, H * (u, v) is the conjugate of H (u, v), P (u, v) is the Fourier transform of the Laplacian, gamma is the boundary blur adjustment coefficient, gamma·|P (u, v) | 2 For reducing the effect of additive noise on the ideal image.
Since the boundary fuzzy adjustment coefficient gamma can be manually adjusted to obtain a proper value, the Fourier transform P (u, v) of the Laplace operator is a constant value; the convolution kernel H (x, y) can be obtained by testing, then the frequency domain expression H (u, v) can be obtained by Fourier transformation, and the H (u, v) can be obtained by taking conjugate * (u, v) and the iToF image sensors of the same process conditions have the same convolution kernel h (x, y); therefore, the regular filter coefficient HP is the same for iToF image sensors of the same process conditions when eliminating crosstalk between pixels.
Specifically, the boundary blur adjustment coefficient γ can be obtained by: the value of gamma is greatly adjusted from small, and whether the image has boundary blurring or not is observed at a test interface; when gamma is increased to a certain first critical value, the image boundary is clear and has no boundary blurring, and when gamma is increased to a certain second critical value, the image boundary blurring occurs again; the gamma value is located in the range from the first threshold value to the second threshold value (including the threshold value itself).
The laplace operator is a two-dimensional isotropic measure of the second spatial derivative of the image, which highlights areas of the image where the intensity changes rapidly, and is therefore commonly used in edge detection tasks. In some embodiments, the laplace operator p (x, y) is expressed using the following formula:
with respect to step S2, the actual image function acquired by the image sensor is acquired. Specifically, the actual image function may be extracted directly from the image sensor. The image of the object to be detected is obtained by shooting an m x n pixel array, each pixel point in the pixel array corresponds to a digital signal, and the digital signal combination is displayed through software, namely the picture seen by people. Thus, the digital signal combination can be extracted directly from the image sensor as a function of the actual image. That is, the actual image function is a discrete array of numbers.
And regarding and carrying out convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficient at S3, obtaining and outputting an ideal image function with crosstalk between pixels eliminated.
In some embodiments, a regular filter coefficient HP is employed as the crosstalk coefficient; then, the step S3 further includes: (1) Converting the crosstalk convolution mathematical model into a crosstalk product mathematical model according to the equivalent relation of the space convolution and the frequency domain element product of Fourier transformation, and converting the crosstalk product mathematical model into an approximate solution solving model of the Fourier transformation function corresponding to the ideal image function according to regular filtering; (2) Acquiring a Fourier transform function corresponding to the actual image function; (3) Calculating the approximate solution solving model according to the Fourier transform function corresponding to the actual image function and the regular filter coefficient to obtain an approximate solution of the Fourier transform function corresponding to the ideal image function; (4) And performing inverse Fourier transform on the approximate solution of the Fourier transform function corresponding to the ideal image function to obtain the approximate solution of the ideal image function.
Specifically, the crosstalk convolution mathematical model is expressed by the above formula 1. Since the convolution in space is equivalent to the product of the frequency domain elements of its fourier transform, namely:
where H (u, v) is the fourier transform of the convolution kernel H (x, y), and F (u, v) is the fourier transform of the ideal image function F (x, y).
The crosstalk convolution mathematical model shown in equation 1 may be converted into a crosstalk product mathematical model in frequency, which is expressed by the following equation:
g (u, v) =H (u, v) ·F (u, v) +N (u, v) (equation 4)
Wherein G (u, v) is the fourier transform corresponding to the actual image function G (x, y), H (u, v) is the fourier transform of the convolution kernel H (x, y), F (u, v) is the fourier transform of the ideal image function F (x, y), and N (u, v) is the fourier transform of the additive noise N (x, y).
And converting the crosstalk product mathematical model into an approximate solution solving model of a Fourier transform function corresponding to the ideal image function according to regular filtering. Specifically, it is obtained from equation 4:
f (u, v) = [ (G (u, v) -N (u, v)) ]/H (u, v) (equation 5)
By adopting regular filtering calculation, an approximate solution solving model of the Fourier transform function corresponding to the ideal image function can be obtained according to the formula 5, and the approximate solution solving model is expressed by adopting the following formula:
wherein ,
and (3) for the approximate solution of the Fourier transform function corresponding to the ideal image function, HP is the regular filter coefficient, and the expression of the regular filter coefficient is shown in formula 2.
In this embodiment, for the image sensor with the same process condition, when eliminating crosstalk between pixels, by acquiring the prestored regular filter coefficient HP and the fourier transform function G (u, v) corresponding to the actual image function acquired by the image sensor, and bringing the regular filter coefficient HP and the fourier transform function G (u, v) corresponding to the actual image function into the above formula 6, the approximate solution of the fourier transform of the ideal image function can be calculated
Then approximate solution
Performing inverse Fourier transform to obtain approximate solution ++of ideal image function f (x, y)>
The crosstalk can be eliminated and the ideal image restored.
The above embodiment of the present invention can eliminate crosstalk and restore an ideal image, but the restoration process of the ideal image requires acquiring and storing a regular filter coefficient HP and calculating an approximate solution of the Fourier transform of the ideal image function according to equation 6
Will approximate solution->
Performing inverse Fourier transform to obtain approximate solution ++of ideal image function f (x, y)>
That is, the calculation is performed in 2 steps, the calculation steps are more, the calculation amount is related to the size of the pixel array, and the larger the pixel array is, the parameters correspond toThe larger the number of columns, the larger the calculation amount, which affects the frame rate of the image sensor.
In order to further optimize an algorithm for eliminating the image crosstalk, the invention further carries out inverse Fourier transform on the regular filter coefficient to obtain a target number sequence as the crosstalk coefficient.
In some embodiments, the target sequence is further obtained by: (1) Performing inverse Fourier transform on the regular filter coefficient HP to obtain an initial sequence d (x, y); (2) A sub-matrix for calculating the central position of the initial sequence d (x, y) according to the calculation requirement
As the target sequence; wherein the initial number sequence d (x, y) is the same for image sensors of the same process conditions when eliminating crosstalk between pixels, the target number sequence (i.e. the sub-matrix +_>
) The larger the approximation solution of the ideal image function is, the closer the ideal image is. The initial number sequence d (x, y) of the image sensor under the same process conditions is the same, and the target number sequence thus obtained +.>
The values are the same, so that the target sequence +.>
And stored in the image sensor to reduce the amount of computation of the image sensor. Target number column->
The larger the calculation amount is, the larger the approximate solution is>
The closer to the ideal image function f (x, y); but the larger the calculation amount, the lower the frame rate of the image sensor. Thus, the crosstalk can be eliminated according to the actualDividing the requirements of precision and frame rate, specifically selecting corresponding target sequence +.>
When a target sequence obtained based on inverse fourier transform of the regular filter coefficient is adopted as the crosstalk coefficient, the step S3 further includes: and converting the crosstalk convolution mathematical model into a space convolution mathematical model according to the equivalence relation of the frequency domain element products of the space convolution and the Fourier transform and the regular filtering, and further calculating the space convolution mathematical model according to the actual image function and the target sequence to obtain the approximate solution of the ideal image function.
In some embodiments, the step of converting the crosstalk convolution mathematical model into the spatial convolution mathematical model according to an equivalence relation of the spatial convolution and a frequency domain element product of the fourier transform and the regular filtering further comprises: (1) Converting the crosstalk convolution mathematical model into a crosstalk product mathematical model according to the equivalent relation of the space convolution and the frequency domain element product of Fourier transformation, and converting the crosstalk product mathematical model into an approximate solution solving model of the Fourier transformation function corresponding to the ideal image function according to regular filtering; (2) And converting the approximate solution model into the space convolution mathematical model according to the equivalence relation of the space convolution and the frequency domain element product of Fourier transformation.
In this embodiment, the crosstalk convolution mathematical model is represented by equation 1, the crosstalk product mathematical model is represented by equation 4, the approximate solution solving model is represented by equation 6, and the regular filter coefficient HP is represented by equation 2. The frequency domain element product represented by equation 6 can be converted into a corresponding spatial convolution, i.e., a spatial convolution mathematical model, according to the equivalence of the spatial convolution represented by equation 3 and its fourier transformed frequency domain element product.
The spatial convolution mathematical model is represented by the following formula:
wherein ,
for an approximate solution of the ideal image function, < > is>
G (x, y) is the actual image function for the target sequence. />
In the present embodiment, for image sensors of the same process conditions, crosstalk between pixels is eliminated by acquiring a pre-stored target sequence
And acquiring an actual image function g (x, y) acquired by the image sensor, and listing the target numbers +.>
And the actual image function g (x, y) is brought into the above equation 7, the approximate solution of the ideal image function can be calculated>
The crosstalk can be eliminated and the ideal image restored. That is, the target sequence +.>
The value can be calculated from other calculation software in advance and stored in the image sensor, crosstalk can be eliminated and an ideal image can be restored by only 1 step of calculation according to the formula 7, the calculated amount of the image sensor is reduced, and the influence on the frame rate of the image sensor is avoided.
Compared with the scheme of eliminating crosstalk between pixels by adopting a deep channel isolation structure in the prior art, the method has the advantages that the influence of the crosstalk is considered in the design process of the image sensor, and the method is inapplicable to the existing image sensor; the method for eliminating crosstalk between pixels of the image sensor provided by the invention can eliminate crosstalk of the existing image sensor, and has a wider application range. The method for eliminating crosstalk between pixels of the image sensor can also be applied to the image sensor after the scheme of eliminating the crosstalk between pixels by adopting the deep channel isolation structure, so that the further elimination of the crosstalk is realized. The method for eliminating crosstalk between pixels of an image sensor provided by the invention is suitable for eliminating crosstalk of an iToF image sensor, and is applicable to other technologies such as: solid-state image sensors such as CMOS image sensors and CDD image sensors are also applicable.
According to the above, the crosstalk problem between pixels is converted into the convolution problem by establishing the crosstalk convolution mathematical model, and for the image sensor with the same process condition, when the crosstalk between pixels is eliminated, the approximate solution of the ideal image function can be obtained by acquiring the prestored regular filter coefficient and the Fourier transform function corresponding to the actual image function acquired by the image sensor through 2-step calculation, and under the condition that the pixel structure is not changed, the crosstalk between pixels is eliminated in a regular filter mode, and the ideal image is restored; or further optimizing an image crosstalk elimination algorithm, obtaining an approximate solution of an ideal image function through 1-step calculation by obtaining a prestored target sequence and obtaining an actual image function acquired by the image sensor, and recovering an ideal image on the basis of reducing the calculated amount of the image sensor.
Based on the same inventive concept, the invention also provides a device for eliminating crosstalk between pixels of the image sensor. The provided device for eliminating the crosstalk between the pixels of the image sensor can adopt the method for eliminating the crosstalk between the pixels of the image sensor as shown in fig. 3 to eliminate the crosstalk between the pixels of the image sensor.
Fig. 5 is a block diagram of a device for eliminating crosstalk between pixels of an image sensor according to an embodiment of the invention. As shown in fig. 5, the device for eliminating crosstalk between pixels of an image sensor includes: a model building module 51, an acquisition module 52 and a processing module 53.
Specifically, the model building module 51 is configured to build a crosstalk convolution mathematical model between an actual image function and an ideal image function of an image sensor pixel, and obtain a regular filter coefficient in advance so as to obtain a crosstalk coefficient and store the crosstalk coefficient; the acquisition module 52 is used for acquiring an actual image function acquired by the image sensor; the processing module 53 is configured to perform convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficient, obtain an ideal image function with crosstalk between pixels eliminated, and output the ideal image function.
In some embodiments, regular filter coefficients are employed as the crosstalk coefficients; the processing module 53 is further configured to: converting the crosstalk convolution mathematical model into a crosstalk product mathematical model according to the equivalent relation of the space convolution and the frequency domain element product of Fourier transformation, and converting the crosstalk product mathematical model into an approximate solution solving model of the Fourier transformation function corresponding to the ideal image function according to regular filtering; acquiring a Fourier transform function corresponding to the actual image function; calculating the approximate solution solving model according to the Fourier transform function corresponding to the actual image function and the regular filter coefficient to obtain an approximate solution of the Fourier transform function corresponding to the ideal image function; and performing inverse Fourier transform on the approximate solution of the Fourier transform function corresponding to the ideal image function to obtain the approximate solution of the ideal image function.
In some embodiments, performing inverse fourier transform on the regular filter coefficients to obtain a target number series as the crosstalk coefficients; the processing module 53 is further configured to: the step of obtaining an ideal image function with crosstalk between pixels eliminated by performing convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficient further comprises: and converting the crosstalk convolution mathematical model into a space convolution mathematical model according to the equivalence relation of the frequency domain element products of the space convolution and the Fourier transform and the regular filtering, and further calculating the space convolution mathematical model according to the actual image function and the target sequence to obtain the approximate solution of the ideal image function.
In some embodiments, the image sensor may be an iToF image sensor, a CMOS image sensor, or a CCD image sensor, or other image sensor that implements imaging via optical imaging principles.
The operation manner of each module may refer to a description of corresponding steps in the method for eliminating crosstalk between pixels of the image sensor shown in fig. 3, which is not repeated herein.
Based on the same inventive concept, the invention also provides an electronic device, comprising a memory, a processor and a computer executable program stored on the memory and capable of running on the processor; the processor, when executing the computer executable program, performs the steps of the method for eliminating crosstalk between pixels of an image sensor as shown in fig. 3.
Within the scope of the inventive concept, embodiments may be described and illustrated in terms of modules that perform one or more of the functions described. These modules (which may also be referred to herein as units, etc.) may be physically implemented by analog and/or digital circuits, for example logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic elements, active electronic elements, optical components, hardwired circuitry, etc., and may optionally be driven by firmware and/or software. The circuitry may be implemented, for example, in one or more semiconductor chips. The circuitry comprising a module may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware that performs some of the functions of the module and a processor that performs other functions of the module. Each module of the embodiments may be physically separated into two or more interacting and discrete modules without departing from the scope of the inventive concept. Likewise, the modules of the embodiments may be physically combined into more complex modules without departing from the scope of the inventive concept.
It should be noted that the terms "comprising" and "having" and their variants are referred to in the document of the present invention and are intended to cover non-exclusive inclusion. The terms "first," "second," and the like are used to distinguish similar objects and not necessarily to describe a particular order or sequence unless otherwise indicated by context, it should be understood that the data so used may be interchanged where appropriate. The term "based on" may be understood as not necessarily intended to express an exclusive set of factors, but may instead, also depend at least in part on the context, allow for other factors to be present that are not necessarily explicitly described. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without collision. In addition, in the above description, descriptions of well-known components and techniques are omitted so as to not unnecessarily obscure the present invention. In the foregoing embodiments, each embodiment is mainly described for differences from other embodiments, and the same/similar parts between the embodiments are referred to each other.
The foregoing is merely illustrative of the preferred embodiments of this invention, and it will be appreciated by those skilled in the art that variations and modifications may be made without departing from the principles of the invention, and such variations and modifications are to be regarded as being within the scope of the invention.
Claims (10)
1. A method for eliminating crosstalk between pixels of an image sensor, comprising:
establishing a crosstalk convolution mathematical model between an actual image function and an ideal image function of an image sensor pixel, acquiring a regular filter coefficient based on a convolution kernel in advance, and further acquiring and storing a crosstalk coefficient;
acquiring an actual image function acquired by an image sensor; and
and carrying out convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficient, obtaining and outputting an ideal image function with crosstalk between pixels eliminated.
2. The method of claim 1, wherein the canonical filter coefficients are further obtained by:
acquiring a convolution kernel through an imaging experiment of a point light source on a central pixel of a pixel array of an image sensor in advance, and acquiring a regular filter coefficient according to the convolution kernel; wherein the regular filter coefficients are the same for image sensors of the same process conditions when eliminating crosstalk between pixels.
3. The method of claim 1, wherein the canonical filter coefficient is represented by the following formula:
wherein HP is the canonical filter coefficient, H (u, v) is the Fourier transform of the convolution kernel, H * (u, v) is the conjugate of H (u, v), P (u, v) is the Fourier transform of the Laplacian, gamma is the boundary blur adjustment coefficient, gamma·|P (u, v) | 2 For reducing the effect of additive noise on the ideal image.
4. A method according to claim 3, characterized in that the laplacian p (x, y) is expressed by the following formula:
5. a method according to claim 3, characterized in that as the crosstalk coefficients, regular filter coefficients are employed;
the step of obtaining an ideal image function with crosstalk between pixels eliminated by performing convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficient further comprises: converting the crosstalk convolution mathematical model into a crosstalk product mathematical model according to the equivalent relation of the space convolution and the frequency domain element product of Fourier transformation, and converting the crosstalk product mathematical model into an approximate solution solving model of the Fourier transformation function corresponding to the ideal image function according to regular filtering;
acquiring a Fourier transform function corresponding to the actual image function;
calculating the approximate solution solving model according to the Fourier transform function corresponding to the actual image function and the regular filter coefficient to obtain an approximate solution of the Fourier transform function corresponding to the ideal image function;
and performing inverse Fourier transform on the approximate solution of the Fourier transform function corresponding to the ideal image function to obtain the approximate solution of the ideal image function.
6. A method according to claim 3, wherein the regular filter coefficients are inverse fourier transformed to obtain a target sequence of numbers as the crosstalk coefficients;
the step of obtaining an ideal image function with crosstalk between pixels eliminated by performing convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficient further comprises: and converting the crosstalk convolution mathematical model into a space convolution mathematical model according to the equivalence relation of the frequency domain element products of the space convolution and the Fourier transform and the regular filtering, and further calculating the space convolution mathematical model according to the actual image function and the target sequence to obtain the approximate solution of the ideal image function.
7. The method of claim 6, wherein the target sequence is further obtained by:
performing inverse Fourier transform on the regular filter coefficient to obtain an initial sequence;
a sub-matrix of the central position of the initial sequence is obtained according to the calculation requirement and is used as the target sequence;
wherein, in eliminating crosstalk between pixels, the initial number columns are the same for image sensors of the same process conditions, the larger the target number columns, the closer the approximation solution of the ideal image function is to the ideal image.
8. The method of claim 6, wherein the step of converting the crosstalk convolution mathematical model to a spatial convolution mathematical model based on an equivalence relation of spatial convolution and a frequency domain element product of a fourier transform and a canonical filtering further comprises:
convolving the crosstalk according to the equivalence relation of the space convolution and the frequency domain element product of Fourier transformation
Converting the mathematical model into a crosstalk product mathematical model, and converting the crosstalk product mathematical model into an approximate solution solving model of a Fourier transform function corresponding to the ideal image function according to regular filtering; and converting the approximate solution model into the space convolution mathematical model according to the equivalence relation of the space convolution and the frequency domain element product of Fourier transformation.
9. The method of claim 8, wherein the step of determining the position of the first electrode is performed,
the crosstalk convolution mathematical model is expressed by the following formula:
g(x，y)＝h(x，y)*f(x，y)+n(x，y)，
wherein g (x, y) is the actual image function, f (x, y) is the ideal image function, h (x, y) is a convolution kernel, and n (x, y) is additive noise;
the crosstalk product mathematical model is expressed by the following formula:
G(u，v)＝H(u，v)·F(u，v)+N(u，v)，
wherein G (u, v) is the fourier transform corresponding to the actual image function G (x, y), H (u, v) is the fourier transform of the convolution kernel H (x, y), F (u, v) is the fourier transform of the ideal image function F (x, y), and N (u, v) is the fourier transform of the additive noise N (x, y); the approximate solution model is represented by the following formula:
wherein ,
an approximate solution of a Fourier transform function corresponding to the ideal image function is obtained, and HP is the regular filter coefficient;
the spatial convolution mathematical model is represented by the following formula:
wherein ,
for an approximate solution of the ideal image function, < > is>
Is the target sequence.
10. An apparatus for eliminating crosstalk between pixels of an image sensor, comprising: the model building module is used for building a crosstalk convolution mathematical model between an actual image function and an ideal image function of the image sensor pixel, acquiring a regular filter coefficient in advance, further acquiring a crosstalk coefficient and storing the crosstalk coefficient;
the acquisition module is used for acquiring an actual image function acquired by the image sensor; and the processing module is used for carrying out convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficient, obtaining and outputting an ideal image function with crosstalk between pixels eliminated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310296645.3A CN116228591A (en) | 2023-03-24 | 2023-03-24 | Method and device for eliminating crosstalk between pixels of image sensor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310296645.3A CN116228591A (en) | 2023-03-24 | 2023-03-24 | Method and device for eliminating crosstalk between pixels of image sensor |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116228591A true CN116228591A (en) | 2023-06-06 |
Family
ID=86591196
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310296645.3A Pending CN116228591A (en) | 2023-03-24 | 2023-03-24 | Method and device for eliminating crosstalk between pixels of image sensor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116228591A (en) |
-
2023
- 2023-03-24 CN CN202310296645.3A patent/CN116228591A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP2899690B1 (en) | Random imaging | |
| Kaur | Noise types and various removal techniques | |
| US7782378B2 (en) | Scene-based non-uniformity correction method using local constant statistics | |
| US9467628B2 (en) | High dynamic range image sensor | |
| KR102391619B1 (en) | Method of infrared image processing for non-uniformity correction | |
| US5925875A (en) | Apparatus and method for compensating for fixed pattern noise in planar arrays | |
| US8229244B2 (en) | Multi-image deblurring | |
| US9719856B2 (en) | Coded spectral imager | |
| Kronander et al. | A unified framework for multi-sensor HDR video reconstruction | |
| US11688039B2 (en) | Method for generating a super-resolution image and related device | |
| CN113228628A (en) | System and method for converting non-bayer pattern color filter array image data | |
| US7876363B2 (en) | Methods, systems and apparatuses for high-quality green imbalance compensation in images | |
| JP2009207118A (en) | Image shooting apparatus and blur correction method | |
| WO2000028730A9 (en) | Adaptive non-uniformity compensation using feedforward shunting and wavelet filter | |
| CN113379636B (en) | Infrared image non-uniformity correction method, device, equipment and storage medium | |
| Sui et al. | Adaptive grayscale adjustment-based stripe noise removal method of single image | |
| JP6344934B2 (en) | Image processing method, image processing apparatus, imaging apparatus, image processing program, and recording medium | |
| US10750135B2 (en) | Hardware-friendly model-based filtering system for image restoration | |
| Bosco et al. | Signal-dependent raw image denoising using sensor noise characterization via multiple acquisitions | |
| CN116228591A (en) | Method and device for eliminating crosstalk between pixels of image sensor | |
| US20090290806A1 (en) | Method and apparatus for the restoration of degraded multi-channel images | |
| Conde et al. | Low-light image enhancement for multiaperture and multitap systems | |
| EP3547251B1 (en) | Dynamic range extension of partially clipped pixels in captured images | |
| Rebiere et al. | Color Pixel Reconstruction for a Monolithic RGB-Z CMOS Imager | |
| Ratliff et al. | Scene-based correction of fixed pattern noise in hyperspectral image data using temporal reordering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20231102 Address after: Room 308 and Room 309, No. 268 Xiangke Road, China (Shanghai) Pilot Free Trade Zone, Pudong New Area, Shanghai, 201210 Applicant after: Shanghai Jieming Technology Co.,Ltd. Address before: 351100 705-706, building a, Jiulong community, No. 999, lihuadong Avenue, Xialin street, Chengxiang District, Putian City, Fujian Province Applicant before: Fujian Jiemu Technology Co.,Ltd. |