CN112233019B - ISP color interpolation method and device based on self-adaptive Gaussian kernel - Google Patents
ISP color interpolation method and device based on self-adaptive Gaussian kernel Download PDFInfo
- Publication number
- CN112233019B CN112233019B CN202011096033.2A CN202011096033A CN112233019B CN 112233019 B CN112233019 B CN 112233019B CN 202011096033 A CN202011096033 A CN 202011096033A CN 112233019 B CN112233019 B CN 112233019B
- Authority
- CN
- China
- Prior art keywords
- gaussian kernel
- channel
- interpolated
- interpolation
- adaptive gaussian
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000004364 calculation method Methods 0.000 claims abstract description 41
- 230000004927 fusion Effects 0.000 claims abstract description 7
- 230000003044 adaptive effect Effects 0.000 claims description 41
- 239000011159 matrix material Substances 0.000 claims description 37
- 238000000926 separation method Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 9
- 230000000694 effects Effects 0.000 abstract description 8
- 230000008901 benefit Effects 0.000 abstract description 5
- 238000012545 processing Methods 0.000 description 20
- 230000008569 process Effects 0.000 description 11
- 230000006870 function Effects 0.000 description 10
- 238000005070 sampling Methods 0.000 description 7
- 238000001914 filtration Methods 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000007781 pre-processing Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000012805 post-processing Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 239000000126 substance Substances 0.000 description 3
- 101100248200 Arabidopsis thaliana RGGB gene Proteins 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 239000003086 colorant Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000012892 rational function Methods 0.000 description 1
- 230000009469 supplementation Effects 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4023—Decimation- or insertion-based scaling, e.g. pixel or line decimation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/60—Rotation of a whole image or part thereof
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20192—Edge enhancement; Edge preservation
Abstract
The invention discloses an ISP color interpolation method and device based on a self-adaptive Gaussian kernel, wherein the method comprises the following steps: s1, acquiring original data of an image to be processed, and separating through RGB channels to respectively obtain R, G, B channel pixels; s2, traversing each pixel point to be interpolated on the G channel, respectively calculating correlation information between each pixel point to be interpolated and a neighborhood, and generating a self-adaptive Gaussian kernel; s3, interpolating each pixel point to be interpolated according to the generated self-adaptive Gaussian kernel, and obtaining an interpolated G channel; s4, interpolation is carried out on the R, B channels respectively to obtain R, B channels after interpolation, and the RGB data after interpolation are obtained after fusion. The invention can sense the edge area and the non-edge area in the image, retain the edge information of the image, and has the advantages of simple implementation method, low calculation complexity, high execution efficiency, good color fidelity effect and the like.
Description
Technical Field
The invention relates to the technical field of ISP (Image Signal Processing ), in particular to an ISP color interpolation method and device based on a self-adaptive Gaussian kernel.
Background
In a digital camera, a single CCD or CMOS image sensor is usually used to collect an image, but the image sensor can only collect one component of RGB colors on a pixel, and 3 image sensors are needed to collect different color components, and considering implementation cost and complexity, a layer of color filter array CFA is usually covered on the surface of the sensor as a filter layer, light filtering points of the filter layer are in one-to-one correspondence with pixels of the image sensor, each light filtering point can only pass through one of red, green and blue lights, that is, a single color component at each pixel coordinate can be obtained through the filter layer, and the other two color components missing at each pixel point are estimated to reconstruct a full-color image, that is, a color interpolation process or an image Demosaic (Demosaic).
The image sensor can provide rich semantic information, and how to more accurately recover an accurate RGB image from the original data output by the image sensor is particularly important, such as in traffic light identification applications. For color interpolation, an ISP method based on bilinear interpolation is generally adopted at present, namely, a convolution kernel is designed for an image scene to be processed, a G channel of an image is convolved by using the convolution kernel, and then an R channel and a B channel are interpolated based on an interpolation result of the G channel. The bilinear interpolation method can be simplified into fixed template convolution, so that parallelization realization is facilitated, the requirements of low calculation amount and high frame rate can be met, but because the equivalent convolution template is irrelevant to image content, obvious aliasing effect (zipper effect) and false color phenomenon (false color) can occur when the image edge is processed. Although the edge saw-tooth effect can be identified and weakened by a post-processing mode, the post-processing mode damages the due flow of the ISP, increases delay links when the image signals are processed in a pipelining mode, and needs to increase a large number of algorithm parameters, so that the workload and difficulty of parameter setting are further increased; on the other hand, the distortion phenomenon caused by the interpolation algorithm is solved by post-processing, which is equivalent to the process that the image structure is destroyed and then reconstructed, and the distortion phenomenon such as saw teeth can be corrected to a certain extent, but the original structure of the image is easier to destroy.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: aiming at the technical problems existing in the prior art, the invention provides the ISP color interpolation method and device based on the self-adaptive Gaussian kernel, which have the advantages of simple implementation method, low calculation complexity, high execution efficiency and good color fidelity effect, and can sense the edge area and the non-edge area in the image and keep the edge information of the image.
In order to solve the technical problems, the technical scheme provided by the invention is as follows:
an ISP color interpolation method based on an adaptive Gaussian kernel comprises the following steps:
s1.rgb channel separation: obtaining original data of an image to be processed, and respectively obtaining R, G, B channel pixels after RGB channel separation;
s2, self-adaptive Gaussian kernel generation: traversing each pixel point to be interpolated on the G channel, respectively calculating correlation information between each pixel point to be interpolated and a neighborhood, and generating an adaptive Gaussian kernel by using the calculated correlation information, wherein the rotation and the expansion of the Gaussian kernel are controlled by using the correlation information;
s3.G channel interpolation: interpolation is carried out on each pixel point to be interpolated according to the generated self-adaptive Gaussian kernel, and a G channel after interpolation is obtained;
s4, RGB channel fusion: and respectively interpolating R, B channels according to the interpolated G channels to obtain interpolated R, B channels, and fusing the interpolated R, G, B channels to obtain interpolated RGB data.
Further, the correlation information is a covariance matrix in the vicinity of the pixel to be interpolated, and the specific step of generating the adaptive gaussian kernel in step S2 includes: and calculating a covariance matrix in the neighborhood of each pixel point to be interpolated, and respectively using each covariance matrix as a matrix for controlling the rotation and the expansion of the Gaussian kernel to generate a corresponding self-adaptive Gaussian kernel.
Further, the covariance matrix is obtained by calculating gradients of pixel points x and y to be interpolated in two directions.
Further, in the step S3, the value of the adaptive gaussian kernel is specifically used to perform weighted average on the neighborhood of each pixel point to be interpolated, so as to obtain interpolation at the position corresponding to each pixel point to be interpolated.
Further, the step S4 includes: and respectively calculating the ratio between the R, B channel and the G channel, and carrying out bilinear interpolation on the R, B channel according to the calculated ratio to obtain the interpolated R, B channel.
Furthermore, an FPGA is adopted to control and execute each step in the color interpolation method according to a flow control mode.
Further, when the FPGA is adopted in a flow control mode, G channel interpolation and RB channel interpolation are sequentially carried out, gradient values of pixel points are calculated by using a plurality of parallelism in the G channel interpolation, then self-adaptive Gaussian kernels are calculated by using a plurality of parallelism based on the calculated gradient values, and then the generated self-adaptive Gaussian kernels are subjected to rational calculation to obtain final interpolation data.
An ISP color interpolation device based on an adaptive gaussian kernel, comprising:
the RGB channel separation module is used for obtaining the original data of the image to be processed, and R, G, B channel pixels are respectively obtained after RGB channel separation;
the self-adaptive Gaussian kernel generation module is used for traversing each pixel point to be interpolated on the G channel, calculating correlation information between each pixel point to be interpolated and a neighborhood respectively, and generating a self-adaptive Gaussian kernel by using the calculated correlation information, wherein the rotation and the expansion of the Gaussian kernel are controlled by using the correlation information;
the G channel interpolation module is used for carrying out interpolation on each pixel point to be interpolated according to the generated self-adaptive Gaussian kernel, so as to obtain an interpolated G channel;
and the RGB channel fusion module is used for respectively interpolating the R, B channels according to the interpolated G channels to obtain the interpolated R, B channels, and fusing the interpolated R, G, B channels to obtain the interpolated RGB data.
An ISP color interpolation device based on an adaptive gaussian kernel, comprising a processor and a memory, said memory being adapted to store a computer program, said processor being adapted to execute said computer program, characterized in that said processor is adapted to execute said computer program to perform the above method.
Compared with the prior art, the invention has the advantages that:
1. according to the invention, in the process of interpolating the G channel pixels, the self-adaptive Gaussian kernel is generated based on the correlation information between the pixel points and the neighborhood, the rotation and the expansion of the Gaussian kernel are controlled by using the correlation information between the pixel points and the neighborhood, and then the interpolation is carried out according to the generated self-adaptive Gaussian kernel, because the rotation and the expansion of each pixel point of the G channel are controlled by using the self-adaptive Gaussian kernel based on the correlation information between the pixel points and the neighborhood, the self-adaptive Gaussian kernel is used for carrying out convolution to adaptively generate different responses on each position of the edge and the non-edge on the image, the directions and the intensities of the edge region and the non-edge region can be perceived after interpolation, so that the edge region in the image can be reserved, the non-edge region can be smoothly processed, the distortion problem can be improved at the source generated by the distortion phenomenon, and the original structure of the image is reserved.
2. The invention further uses the covariance matrix as a matrix for controlling the rotation and the extension of the Gaussian kernel, and the intensity and the direction of the edge can be reflected based on the covariance matrix, so that the energy of the Gaussian kernel is concentrated on the area of the edge of the image, thereby being capable of adaptively keeping the edge area, avoiding the exponential calculation based on natural logarithm, accelerating the processing speed, and filtering the image while keeping the edge information.
3. The invention further rationalizes the self-adaptive Gaussian kernel function based on the interpolation, can greatly reduce the calculated amount of color interpolation and accelerate the interpolation processing speed, and is convenient to be applied to FPGA platforms based on the rational processing mode, thereby effectively improving the processing efficiency and reducing the consumption of resources.
4. According to the invention, the pseudo-color effect can be reduced by further interpolating and supplementing the sampling frequencies of the R channel and the B channel, and meanwhile, the information exchange between the three color channels is increased in the interpolation process of the R channel and the B channel by carrying out bilinear interpolation on the R, B channel in a proportional manner based on the G channel, so that the influence caused by different sampling frequencies on the RGB three channels can be counteracted.
Drawings
Fig. 1 is a schematic diagram of an implementation flow of an ISP color interpolation method based on an adaptive gaussian kernel in this embodiment.
Fig. 2 is a schematic diagram of the principle of using gaussian kernels with adaptive gaussian kernels.
FIG. 3 is a detailed flow chart of color interpolation based on adaptive Gaussian kernel verification in an embodiment of the invention.
FIG. 4 is a schematic diagram of a particular pipeline control for implementing color interpolation based on an FPGA in a particular application embodiment of the present invention.
Detailed Description
The invention is further described below in connection with the drawings and the specific preferred embodiments, but the scope of protection of the invention is not limited thereby.
As shown in fig. 1, the steps of the ISP color interpolation method based on the adaptive gaussian kernel of the present embodiment include:
s1.rgb channel separation: obtaining original data of an image to be processed, and respectively obtaining R, G, B channel pixels after RGB channel separation;
s2, self-adaptive Gaussian kernel generation: traversing each pixel point to be interpolated on the G channel, respectively calculating correlation information between each pixel point to be interpolated and a neighborhood, and generating an adaptive Gaussian kernel by using the calculated correlation information, wherein the rotation and the expansion of the Gaussian kernel are controlled by using the correlation information;
s3.G channel interpolation: interpolation is carried out on each pixel point to be interpolated according to the generated self-adaptive Gaussian kernel, and a G channel after interpolation is obtained;
s4, RGB channel fusion: and respectively interpolating R, B channels according to the interpolated G channels to obtain interpolated R, B channels, and fusing the interpolated R, G, B channels to obtain interpolated RGB data.
The gaussian kernel is generally used to up-sample and filter the image, and the energy distribution of a typical gaussian kernel is shown in fig. 2 (a), where the origin of the coordinate system represents the center of the gaussian kernel neighborhood, and the closer to the center the higher the energy, i.e., the greater the weight of the pixel there. The edge information can be lost by directly adopting Gaussian kernel to carry out up-sampling and filtering, the sharpening degree of the image is reduced, and when the edge of the image is processed, the edge information can be reserved while the image is up-sampled by further improving the weight at the edge. Assuming that the shape of the edge is shown in fig. 2 (b), where the dot represents the target pixel, to achieve upsampling of the image while preserving the edge information, it is necessary to construct a gaussian kernel as shown in fig. 2 (c) under the energy profile.
According to the embodiment, the characteristics of the Gaussian kernel function are considered, the self-adaptive Gaussian kernel is generated based on the correlation information between the pixel points and the neighborhood in the G channel pixel interpolation process, the rotation and the expansion of the Gaussian kernel are controlled by using the correlation information between the pixel points and the neighborhood, the interpolation is carried out according to the generated self-adaptive Gaussian kernel, the rotation and the expansion are controlled by the self-adaptive Gaussian kernel based on the correlation information between the pixel points and the neighborhood, so that different responses are self-adaptively generated on each position of the edge and the non-edge on the image by using the self-adaptive Gaussian kernel in the convolution, the directions and the intensities of the edge region and the non-edge region can be perceived after interpolation, the edge region in the image can be reserved, the non-edge region can be smoothed, meanwhile, the distortion problem can be improved at the source generated by the distortion phenomenon, and the original structure of the image can be reserved.
In this embodiment, the correlation information is specifically a covariance matrix in the vicinity of the pixel to be interpolated, and the specific step of generating the adaptive gaussian kernel in step S2 includes: and calculating covariance matrixes in the neighborhood of each pixel point to be interpolated, and respectively using each covariance matrix as a matrix for controlling rotation and expansion of the Gaussian kernel to generate a corresponding self-adaptive Gaussian kernel. The correlation between the pixel points and the neighborhood pixel points can be effectively represented based on the covariance matrix, so that the covariance matrix is used as a matrix for controlling the rotation and the expansion of the Gaussian kernel, and the generated adaptive Gaussian kernel can accurately and adaptively sense the intensity and the direction of the edge region, so that the edge region can be accurately reserved.
The typical gaussian kernel function calculation formula is specifically:
wherein x and y are respectively the abscissa and ordinate of the pixel point in the image, h is the standard deviation of the Gaussian kernel, the concentration degree of the Gaussian kernel weight can be controlled by adjusting the standard deviation h, and the smaller the standard deviation is, the larger the pixel of the central part occupies the weight, and otherwise, the smaller the weight is.
Writing the Gaussian kernel function into a matrix form is as follows:
wherein the method comprises the steps of
To control the matrix of gaussian kernel rotation and expansion, the control matrix is used to realize rotation and expansion only at specific positions.
In this embodiment, the covariance matrix is specifically obtained by calculating gradients in the x and y directions of the pixel points to be interpolated, which is
As shown in fig. 2 (d), the calculation formula of the adaptive gaussian kernel is:
wherein K (x, y) is an adaptive Gaussian kernel of the pixel point (x, y), C (x,y) Is the covariance moment in the neighborhood of the pixel (x, y).
Covariance matrix C (x,y) The intensity and direction of the edges can be reflected, in this embodiment by using the covariance matrix C (x,y) In response to the edges of the image, the Gaussian kernel rotation and expansion are controlled to concentrate the energy of the Gaussian kernel on the area of the edges of the image, namelyThe edge region adopts self-adaptive high weight, so that the edge region can be reserved in a self-adaptive mode.
It is understood that, besides the adaptive gaussian kernel shown in the above formula (1), other adaptive gaussian kernels may be adopted according to actual requirements.
The traditional Gaussian kernel filtering-based index calculation based on natural logarithm is low in calculation efficiency, and is difficult to meet the real-time requirement of a camera carried on an unmanned vehicle, and when the method is directly applied to color interpolation, the method is insensitive to edge information in an image, and an interpolation result is fuzzy globally. According to the method, the covariance matrix in the pixel point neighborhood is calculated, the self-adaptive Gaussian kernel is generated based on the covariance matrix to conduct interpolation, and the exponential calculation with the natural logarithm as the base can be avoided, so that the processing speed can be increased, and the image can be filtered while the edge information is kept.
In step S2 of this embodiment, the method further includes performing a physical and chemical calculation on the generated adaptive gaussian kernel, so as to obtain a final physical and chemical adaptive gaussian kernel output. The rationalization of the function is to perform rational approximation on the function, the rational function is adopted to represent the original function, a great amount of calculation still needs to be executed by directly using the self-adaptive Gaussian kernel to perform interpolation, and the embodiment further can greatly reduce the calculated amount of color interpolation and accelerate the interpolation processing speed by rationalizing the self-adaptive Gaussian kernel function on the basis of performing interpolation on the basis of the self-adaptive Gaussian kernel function, and can be conveniently applied to an FPGA platform on the basis of rationalization processing mode, so that the processing efficiency of the platform can be effectively improved and the consumption of resources is reduced.
In step S3 of this embodiment, the value of the adaptive gaussian kernel is specifically used to perform weighted average on the neighborhood of each pixel to be interpolated, so as to obtain the interpolation at the position corresponding to each pixel to be interpolated. Of course, other statistical calculation methods can be adopted to obtain final interpolation, such as weighting method, according to actual requirements.
The step of step S4 in this embodiment includes: and respectively calculating the ratio between the R, B channel and the G channel, and carrying out bilinear interpolation on the R, B channel according to the calculated ratio to obtain an interpolated R, B channel. Conventional ISP color interpolation algorithms based on bilinear interpolation need to be limited by the difference of sampling frequencies of different color channels, the photosensitive elements of the embedded image sensor are usually arranged in RGGB or BGGR, and the sampling frequency on the G channel is twice that of the R channel and the B channel, so that analysis of the color interpolation algorithm based on bilinear interpolation results in local dislocation of the RGB three channels in space. According to the embodiment, the false color effect can be reduced by interpolation and supplementation of sampling frequencies of the R channel and the B channel, meanwhile, bilinear interpolation is carried out on the R, B channel in a proportional mode based on the G channel, information communication among three color channels is increased in the interpolation process of the R channel and the B channel, and influences caused by different sampling frequencies on the RGB three channels can be offset.
As shown in fig. 3, the process of implementing color interpolation in the specific application embodiment of the present invention is as follows:
step 1: after preprocessing such as dead pixel detection is carried out on the input original Bayer grid data, RGGB channels of the original Bayer grid are separated, and R, G, B channels are separated;
step 2: pixel traversal is carried out on the separated G channel, and covariance matrixes in the neighborhood of a specified range (such as a specific preferable 5*5) of each pixel point to be interpolated are calculated;
step 3: calculating a rational self-adaptive Gaussian kernel according to the covariance matrix of each pixel neighborhood;
step 4: carrying out weighted average on the neighborhood by utilizing the value in the rational self-adaptive Gaussian kernel to obtain interpolation of the position of each pixel point;
step 5: calculating the ratio of R, B channel to G channel, wherein only the part with value in R, B channel is calculated;
step 6: performing bilinear interpolation on the R, B channel by combining the ratio of the R, B channel to the G channel to obtain an interpolation result of the R, B channel;
step 7: and fusing interpolation results of all the channels to output RGB data.
In this embodiment, the steps in the color interpolation method are controlled and executed by using the FPGA according to the pipeline control manner, so that the whole processing process uses the pipeline processing manner, so that the interpolation processing speed and the data throughput are further improved by using the advantage of parallel processing of the FPGA.
In this embodiment, when the FPGA is used in a flow control manner, the G channel interpolation and the RB channel interpolation are sequentially performed, where the G channel interpolation is divided into a gradient calculation, a covariance matrix calculation, a weight calculation, and a multi-stage flow of the G channel interpolation calculation, the gradient is calculated by the gradient calculation stage flow, the covariance matrix is calculated by the covariance matrix calculation stage flow based on the calculated gradient, the weight required by the interpolation is calculated by the weight calculation stage flow, and then the calculation of the physicochemical adaptive gaussian kernel is completed by the G channel interpolation stage flow, so as to obtain a final G channel interpolation result. In the FPGA pipeline control process, the G channel interpolation is calculated based on the rational self-adaptive Gaussian kernel function mode, so that interpolation processing can be efficiently executed on an FPGA platform, and meanwhile, the resource consumption required in the interpolation process can be reduced.
In a specific application embodiment, each step of the color interpolation method is divided into multiple stages of pipeline, and the steps which can be executed in parallel are configured to be executed in parallel, as shown in fig. 4, each stage of pipeline is specifically divided into:
(1) G, preprocessing interpolation data: the data blocks required for gradient computation are generated and the image edges are preprocessed. Generating 7*7 data blocks required by gradient calculation by using a tap method, processing the edges of the image, and performing cross-clock domain processing on the processed data;
(2) Gradient calculation: the gradient values in the x and y directions of the non-green pixel points of the pixel block are calculated using a plurality of parallelism. The gradient values in the x and y directions of 13/12 non-green pixel points in the 5*5 pixel block are calculated twice by using 7 parallelism, and the gradient values can be configured according to actual requirements;
(3) Covariance matrix calculation: calculating the product of gradients by using a plurality of parallelism degrees, and respectively calculating the square of the x gradient, the square of the y gradient and the product of the two gradients; the obtained products are grouped and accumulated. Specifically, 7 parallelism degrees are used for calculating the product of the gradients, each parallelism degree comprises three integer multipliers, and the square of the x gradient and the square of the y gradient are calculated respectively; the 7 obtained multiplications are divided into two groups, one group is added by 4 numbers, the other group is added by 3 numbers, and finally the two sums are accumulated;
(4) Weight calculation: the packet performs weight calculation. The calculated weight is changed into actual half (7/6) because of the weight symmetry, wherein the weight is equal to zero at the center of the 5*5 pixel block, so that no calculation is performed; dividing the calculation time into two groups, and calculating three weights each time;
(5) G channel interpolation calculation: the method comprises the steps of performing interpolation calculation in two steps, wherein the whole calculation uses a pipeline mode, firstly summing weights and the green pixel point multiplication respectively, and then performing division operation to finish the calculation of a rational self-adaptive Gaussian kernel, so as to calculate interpolation data;
(6) G interpolation synchronous output: outputting two paths of data, wherein one path is original image data without interpolation and the other path is image data after G channel interpolation, and the two paths of data are synchronously output after cross-clock domain processing;
(7) Preprocessing RB channel interpolation data: generating a data block required for interpolation and preprocessing the image edge. Generating 3*3 data blocks required by interpolation by using a tap method, processing the edges of the image, and performing cross-clock domain processing on the processed data;
(8) RB interpolation: calculating interpolation data of R and B channels by using the original data and the interpolated G channel data, and calculating interpolation of R and B by using the same logic resource as RB alternately appears;
(9) RB interpolation synchronous output: and performing cross-clock domain processing on the interpolated RB data and the interpolated G channel data, and then synchronously outputting RGB component data.
Through the above-mentioned pipelining division, the parallel advantage of the FPGA platform can be fully exerted, the processes which can be processed in parallel are performed in parallel as much as possible, the color interpolation can be realized efficiently, and meanwhile, the interpolation effect is ensured.
It can be understood that, besides the above-mentioned pipeline dividing mode executed by the FPGA, other pipeline dividing modes and parallel configuration modes can be adopted according to actual requirements.
The embodiment also provides an ISP color interpolation device based on the adaptive gaussian kernel, which comprises:
the RGB channel separation module is used for obtaining the original data of the image to be processed, and R, G, B channel pixels are respectively obtained after RGB channel separation;
the self-adaptive Gaussian kernel generation module is used for traversing each pixel point to be interpolated on the G channel, calculating the correlation information between each pixel point to be interpolated and the neighborhood respectively, and generating a self-adaptive Gaussian kernel by using the calculated correlation information, wherein the rotation and the expansion of the Gaussian kernel are controlled by using the correlation information;
the G channel interpolation module is used for carrying out interpolation on each pixel point to be interpolated according to the generated self-adaptive Gaussian kernel, so as to obtain an interpolated G channel;
and the RGB channel fusion module is used for respectively interpolating the R, B channels according to the interpolated G channels to obtain the interpolated R, B channels, and fusing the interpolated R, G, B channels to obtain the interpolated RGB data.
The ISP color interpolation device based on the adaptive gaussian kernel in this embodiment corresponds to the ISP color interpolation method based on the adaptive gaussian kernel one by one, and will not be described in detail here.
In another embodiment, the ISP color interpolation device based on the adaptive gaussian kernel may further be: the method comprises a processor and a memory, wherein the memory is used for storing a computer program, the processor is used for executing the computer program, and the processor is used for executing the computer program to execute the ISP color interpolation method based on the adaptive Gaussian kernel.
The foregoing is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. While the invention has been described with reference to preferred embodiments, it is not intended to be limiting. Therefore, any simple modification, equivalent variation and modification of the above embodiments according to the technical substance of the present invention shall fall within the scope of the technical solution of the present invention.
Claims (10)
1. An ISP color interpolation method based on an adaptive Gaussian kernel is characterized by comprising the following steps:
s1.rgb channel separation: obtaining original data of an image to be processed, and respectively obtaining R, G, B channel pixels after RGB channel separation;
s2, self-adaptive Gaussian kernel generation: traversing each pixel point to be interpolated on the G channel, respectively calculating correlation information between each pixel point to be interpolated and a neighborhood, and generating an adaptive Gaussian kernel by using the calculated correlation information, wherein the rotation and the expansion of the Gaussian kernel are controlled by using the correlation information, and the correlation information is a covariance matrix;
s3.G channel interpolation: interpolation is carried out on each pixel point to be interpolated according to the generated self-adaptive Gaussian kernel, and a G channel after interpolation is obtained;
s4, RGB channel fusion: and respectively interpolating R, B channels according to the interpolated G channels to obtain interpolated R, B channels, and fusing the interpolated R, G, B channels to obtain interpolated RGB data.
2. The ISP color interpolation method based on adaptive gaussian kernel according to claim 1, wherein the correlation information is covariance matrix in the neighborhood of the pixel to be interpolated, and the specific step of generating the adaptive gaussian kernel in step S2 comprises: and calculating a covariance matrix in the neighborhood of each pixel point to be interpolated, and respectively using each covariance matrix as a matrix for controlling the rotation and the expansion of the Gaussian kernel to generate a corresponding self-adaptive Gaussian kernel.
3. The adaptive gaussian kernel-based ISP color interpolation method according to claim 2, wherein: the covariance matrix is obtained by gradient calculation in the x direction and the y direction of the pixel points to be interpolated.
4. The ISP color interpolation method based on adaptive gaussian kernel according to claim 1, wherein in step S2, the method further comprises performing a rational computation on the generated adaptive gaussian kernel to obtain a final rational adaptive gaussian kernel output.
5. The ISP color interpolation method according to any one of claims 1 to 4, wherein in the step S3, the pixel neighborhood to be interpolated is weighted and averaged by using the value of the adaptive gaussian kernel, so as to obtain the interpolation at the position corresponding to each pixel to be interpolated.
6. The ISP color interpolation method based on adaptive gaussian kernel according to any one of claims 1 to 4, wherein said step S4 comprises: and respectively calculating the ratio between the R, B channel and the G channel, and carrying out bilinear interpolation on the R, B channel according to the calculated ratio to obtain the interpolated R, B channel.
7. The ISP color interpolation method based on adaptive gaussian kernel according to any one of claims 1 to 4, wherein each step in the color interpolation method is controlled and executed in a pipeline control manner by using an FPGA.
8. The ISP color interpolation method based on adaptive gaussian kernel according to claim 7, wherein the G channel interpolation and the RB channel interpolation are sequentially performed according to a flow control manner by using the FPGA, wherein the G channel interpolation is divided into a plurality of steps of flow of gradient calculation, covariance matrix calculation, weight calculation and G channel interpolation calculation, the gradient is calculated by the gradient calculation step flow, the covariance matrix is calculated by the covariance matrix calculation step flow based on the calculated gradient, the weight required for interpolation is calculated by the weight calculation step flow, and the calculation of the rational adaptive gaussian kernel is completed by the G channel interpolation step flow, thereby obtaining the final G channel interpolation result.
9. An ISP color interpolation device based on an adaptive gaussian kernel, comprising:
the RGB channel separation module is used for obtaining the original data of the image to be processed, and R, G, B channel pixels are respectively obtained after RGB channel separation;
the self-adaptive Gaussian kernel generation module is used for traversing each pixel point to be interpolated on the G channel, calculating correlation information between each pixel point to be interpolated and a neighborhood respectively, and generating a self-adaptive Gaussian kernel by using the calculated correlation information, wherein the rotation and the expansion of the Gaussian kernel are controlled by using the correlation information, and the correlation information is a covariance matrix;
the G channel interpolation module is used for carrying out interpolation on each pixel point to be interpolated according to the generated self-adaptive Gaussian kernel, so as to obtain an interpolated G channel;
and the RGB channel fusion module is used for respectively interpolating the R, B channels according to the interpolated G channels to obtain the interpolated R, B channels, and fusing the interpolated R, G, B channels to obtain the interpolated RGB data.
10. An ISP color interpolation device based on an adaptive gaussian kernel, comprising a processor and a memory, the memory being adapted to store a computer program, the processor being adapted to execute the computer program, characterized in that the processor is adapted to execute the computer program to perform the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011096033.2A CN112233019B (en) | 2020-10-14 | 2020-10-14 | ISP color interpolation method and device based on self-adaptive Gaussian kernel |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011096033.2A CN112233019B (en) | 2020-10-14 | 2020-10-14 | ISP color interpolation method and device based on self-adaptive Gaussian kernel |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112233019A CN112233019A (en) | 2021-01-15 |
| CN112233019B true CN112233019B (en) | 2023-06-30 |
Family
ID=74112785
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011096033.2A Active CN112233019B (en) | 2020-10-14 | 2020-10-14 | ISP color interpolation method and device based on self-adaptive Gaussian kernel |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112233019B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115063460A (en) * | 2021-12-24 | 2022-09-16 | 山东建筑大学 | High-precision self-adaptive homonymous pixel interpolation and optimization method |
| CN115689887A (en) * | 2022-10-28 | 2023-02-03 | 辉羲智能科技(上海)有限公司 | Automatic driving RGBIR image resampling method, system, terminal and medium |
| CN115689895B (en) * | 2023-01-01 | 2023-04-07 | 武汉图科智能科技有限公司 | Up-sampling hardware processing system based on Lanczos interpolation |
| CN116912096B (en) * | 2023-09-14 | 2024-01-09 | 惠州市耀盈精密技术有限公司 | IMOLD system-based precision mold template image generation method |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7412429B1 (en) * | 2007-11-15 | 2008-08-12 | International Business Machines Corporation | Method for data classification by kernel density shape interpolation of clusters |
| EP2124187A1 (en) * | 2008-05-22 | 2009-11-25 | Telefonaktiebolaget LM Ericsson (PUBL) | Apparatus and method for demosaicing |
| CN103686111A (en) * | 2013-12-31 | 2014-03-26 | 上海富瀚微电子有限公司 | Method and device for correcting color based on RGBIR (red, green and blue, infra red) image sensor |
| CN103995860A (en) * | 2014-05-16 | 2014-08-20 | 中国科学院重庆绿色智能技术研究院 | Image retrieval system and method based on color feature extraction |
| CN104794691A (en) * | 2015-04-07 | 2015-07-22 | 浙江大学 | Definition reconstruction method of single out-of-focus image using generalized Gaussian model |
| EP3051484A2 (en) * | 2015-01-30 | 2016-08-03 | BOE Technology Group Co., Ltd. | Image interpolation method and image interpolation apparatus |
| CN106910168A (en) * | 2017-01-09 | 2017-06-30 | 中国科学院自动化研究所 | Parallel image color enhancement method and apparatus |
| CN107403435A (en) * | 2017-07-28 | 2017-11-28 | 江苏大学 | A kind of Color Image Edge extracting method based on RGB color |
| CN110706174A (en) * | 2019-09-27 | 2020-01-17 | 集美大学 | Image enhancement method, terminal equipment and storage medium |
| JP2020155907A (en) * | 2019-03-20 | 2020-09-24 | 株式会社朋栄 | Automatic color correction processing method and automatic color correction processing device |
-
2020
- 2020-10-14 CN CN202011096033.2A patent/CN112233019B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7412429B1 (en) * | 2007-11-15 | 2008-08-12 | International Business Machines Corporation | Method for data classification by kernel density shape interpolation of clusters |
| EP2124187A1 (en) * | 2008-05-22 | 2009-11-25 | Telefonaktiebolaget LM Ericsson (PUBL) | Apparatus and method for demosaicing |
| CN103686111A (en) * | 2013-12-31 | 2014-03-26 | 上海富瀚微电子有限公司 | Method and device for correcting color based on RGBIR (red, green and blue, infra red) image sensor |
| CN103995860A (en) * | 2014-05-16 | 2014-08-20 | 中国科学院重庆绿色智能技术研究院 | Image retrieval system and method based on color feature extraction |
| EP3051484A2 (en) * | 2015-01-30 | 2016-08-03 | BOE Technology Group Co., Ltd. | Image interpolation method and image interpolation apparatus |
| CN104794691A (en) * | 2015-04-07 | 2015-07-22 | 浙江大学 | Definition reconstruction method of single out-of-focus image using generalized Gaussian model |
| CN106910168A (en) * | 2017-01-09 | 2017-06-30 | 中国科学院自动化研究所 | Parallel image color enhancement method and apparatus |
| CN107403435A (en) * | 2017-07-28 | 2017-11-28 | 江苏大学 | A kind of Color Image Edge extracting method based on RGB color |
| JP2020155907A (en) * | 2019-03-20 | 2020-09-24 | 株式会社朋栄 | Automatic color correction processing method and automatic color correction processing device |
| CN110706174A (en) * | 2019-09-27 | 2020-01-17 | 集美大学 | Image enhancement method, terminal equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 基于RGB空间剖分的彩色图像边缘检测;赵景秀;王菁;赵昭;;光电子技术(第03期) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112233019A (en) | 2021-01-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112233019B (en) | ISP color interpolation method and device based on self-adaptive Gaussian kernel | |
| Chakrabarti | Learning sensor multiplexing design through back-propagation | |
| JP6012375B2 (en) | Pixel interpolation processing device, imaging device, program, and integrated circuit | |
| US7391904B2 (en) | Correlation-based color mosaic interpolation adjustment using luminance gradients | |
| EP2523160A1 (en) | Image processing device, image processing method, and program | |
| JP2000341707A (en) | Method for releasing mosaic of image by using directional smoothing operation | |
| CN102254301B (en) | Demosaicing method for CFA (color filter array) images based on edge-direction interpolation | |
| EP3924933A1 (en) | Image processor | |
| CN113628146B (en) | Image denoising method based on depth convolution network | |
| JP2003143615A (en) | Apparatus and method for processing output data from image sensor | |
| CN111340732B (en) | Low-illumination video image enhancement method and device | |
| CN1968422A (en) | Color interpolation apparatus and method for reconstructing missing colors utilizing image edge indicators | |
| CN104751415A (en) | Image denoising and enhancing method and device and image processing system | |
| CN112465727A (en) | Low-illumination image enhancement method without normal illumination reference based on HSV color space and Retinex theory | |
| CN104751416A (en) | Image denoising method and device and image processing system | |
| CN110852953A (en) | Image interpolation method and device, storage medium, image signal processor and terminal | |
| CN113034391A (en) | Multi-mode fusion underwater image enhancement method, system and application | |
| CN111008936A (en) | Multispectral image panchromatic sharpening method | |
| CN113068011B (en) | Image sensor, image processing method and system | |
| CN115004220A (en) | Neural network for raw low-light image enhancement | |
| WO2022061879A1 (en) | Image processing method, apparatus and system, and computer-readable storage medium | |
| US20220130012A1 (en) | Demosaicing method and demosaicing device | |
| CN104200214A (en) | Method for describing image local feature areas by walsh converted power spectrums | |
| CN116917954A (en) | Image detection method and device and electronic equipment | |
| EP4339877A1 (en) | Method and apparatus for super resolution |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Guo Wang Inventor after: Wu Qingqing Inventor after: Hu Tingbo Inventor after: An Xiangjing Inventor after: Zhang Mengxuan Inventor before: Guo Wang Inventor before: Hu Tingbo Inventor before: An Xiangjing Inventor before: Zhang Mengxuan |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |