CN112233019A - ISP color interpolation method and device based on self-adaptive Gaussian kernel - Google Patents
ISP color interpolation method and device based on self-adaptive Gaussian kernel Download PDFInfo
- Publication number
- CN112233019A CN112233019A CN202011096033.2A CN202011096033A CN112233019A CN 112233019 A CN112233019 A CN 112233019A CN 202011096033 A CN202011096033 A CN 202011096033A CN 112233019 A CN112233019 A CN 112233019A
- Authority
- CN
- China
- Prior art keywords
- channel
- gaussian kernel
- interpolation
- interpolated
- adaptive gaussian
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000004364 calculation method Methods 0.000 claims abstract description 47
- 238000000926 separation method Methods 0.000 claims abstract description 13
- 230000004927 fusion Effects 0.000 claims abstract description 7
- 230000003044 adaptive effect Effects 0.000 claims description 50
- 239000011159 matrix material Substances 0.000 claims description 39
- 238000004590 computer program Methods 0.000 claims description 9
- 230000008901 benefit Effects 0.000 abstract description 6
- 230000000694 effects Effects 0.000 abstract description 6
- 238000012545 processing Methods 0.000 description 14
- 230000008569 process Effects 0.000 description 11
- 230000006870 function Effects 0.000 description 8
- 239000000047 product Substances 0.000 description 7
- 238000005070 sampling Methods 0.000 description 7
- 238000007781 pre-processing Methods 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000001914 filtration Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000012805 post-processing Methods 0.000 description 3
- 101100248200 Arabidopsis thaliana RGGB gene Proteins 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000036571 hydration Effects 0.000 description 1
- 238000006703 hydration reaction Methods 0.000 description 1
- 238000012892 rational function Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4023—Scaling of whole images or parts thereof, e.g. expanding or contracting based on decimating pixels or lines of pixels; based on inserting pixels or lines of pixels
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/60—Rotation of whole images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20192—Edge enhancement; Edge preservation
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an ISP color interpolation method and a device based on a self-adaptive Gaussian kernel, wherein the method comprises the following steps: s1, acquiring original data of an image to be processed, and respectively obtaining R, G, B channel pixels after RGB channel separation; s2, traversing each pixel point to be interpolated on the G channel, respectively calculating correlation information between each pixel point to be interpolated and a neighborhood, and generating a self-adaptive Gaussian kernel; s3, interpolating each pixel point to be interpolated according to the generated self-adaptive Gaussian core to obtain a G channel after interpolation; and S4, respectively interpolating R, B channels to obtain R, B channels after interpolation, and obtaining RGB data after interpolation after fusion. The invention can sense the edge area and the non-edge area in the image, retains the edge information of the image, and has the advantages of simple realization method, low calculation complexity, high execution efficiency, good color fidelity effect and the like.
Description
Technical Field
The present invention relates to the field of ISP (Image Signal Processing) technology, and in particular, to an ISP color interpolation method and apparatus based on an adaptive gaussian kernel.
Background
In a digital camera, a single CCD or CMOS image sensor is usually used to realize image acquisition, but the image sensor can only acquire one component of RGB colors on one pixel, and 3 image sensors are required to acquire different color components, in consideration of implementation cost and complexity, a color filter array CFA is usually covered on the sensor surface as a filter layer, the filter points of the filter layer correspond to the pixels of the image sensor one to one, each filter point can only pass one of red, green, and blue lights, that is, a single color component at each pixel coordinate can be obtained through the filter layer, and the process of reconstructing one full-color image by estimating the other two missing color components at each pixel point is a color interpolation process or an image Demosaic (Demosaic).
The image sensor can provide rich semantic information, and it is important to more accurately recover an accurate RGB image from raw data output by the image sensor, such as in a traffic light recognition application. For color interpolation, at present, an ISP method based on bilinear interpolation is usually adopted, that is, a convolution kernel is designed for an image scene to be processed, the G channel of the image is convolved by using the convolution kernel, and then the R channel and the B channel are interpolated based on an interpolation result of the G channel. The bilinear interpolation method can be simplified into fixed template convolution, so that parallelization is convenient to realize, and the requirements of low calculation amount and high frame rate can be met. Although the edge sawtooth effect can be identified and weakened through a post-processing mode, the post-processing mode destroys an ISP due flow, adds a delay link when processing an image signal in a hydration mode, needs to add a large number of algorithm parameters and further increases the workload and difficulty of parameter setting; on the other hand, the distortion phenomenon caused by the interpolation algorithm is solved by post-processing, which is equivalent to that the image structure is firstly destroyed and then reconstructed, although the distortion phenomena such as sawtooth and the like can be corrected to a certain degree, the original structure of the image is easier to be destroyed.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: aiming at the technical problems in the prior art, the invention provides the ISP color interpolation method and the device based on the self-adaptive Gaussian kernel, which have the advantages of simple implementation method, low calculation complexity, high execution efficiency and good color fidelity effect, and can sense the edge area and the non-edge area in the image and reserve the edge information of the image.
In order to solve the technical problems, the technical scheme provided by the invention is as follows:
an ISP color interpolation method based on an adaptive Gaussian kernel comprises the following steps:
s1, RGB channel separation: acquiring original data of an image to be processed, and respectively obtaining R, G, B channel pixels after RGB channel separation;
s2, self-adaptive Gaussian kernel generation: traversing each pixel point to be interpolated on the G channel, respectively calculating correlation information between each pixel point to be interpolated and a neighborhood, and generating a self-adaptive Gaussian kernel by using the correlation information obtained by calculation, wherein the correlation information is used for controlling the rotation and the expansion of the Gaussian kernel;
s3.G channel interpolation: interpolating each pixel point to be interpolated according to the generated self-adaptive Gaussian core to obtain an interpolated G channel;
s4, RGB channel fusion: and respectively interpolating R, B channels according to the interpolated G channels to obtain R, B channels after interpolation, and fusing R, G, B channels after interpolation to obtain RGB data after interpolation.
Further, the correlation information is a covariance matrix in a neighborhood of a pixel to be interpolated, and the specific step of generating the adaptive gaussian kernel in step S2 includes: and calculating a covariance matrix in the neighborhood of each pixel point to be interpolated, and generating a corresponding self-adaptive Gaussian kernel by using each covariance matrix as a matrix for controlling rotation and expansion of the Gaussian kernel.
Further, the covariance matrix is obtained by calculating gradients of the pixel points to be interpolated in the x and y directions.
Further, in step S3, the weighted average is specifically performed on the neighborhood of each pixel to be interpolated by using the value of the adaptive gaussian kernel, so as to obtain the interpolation corresponding to the position of each pixel to be interpolated.
Further, the step of step S4 includes: and respectively calculating the ratio between the R, B channel and the G channel, and performing bilinear interpolation on the R, B channel according to the calculated ratio to obtain a R, B channel after interpolation.
Furthermore, the FPGA is adopted to control and execute each step in the color interpolation method according to a flow control mode.
Furthermore, when the FPGA is used according to a pipeline control method, G channel interpolation and RB channel interpolation are performed in sequence, in the G channel interpolation, a plurality of parallelism degrees are used to calculate gradient values of pixel points, then a plurality of parallelism degrees are used to calculate an adaptive gaussian kernel based on the calculated gradient values, and then physicochemical calculation is performed on the generated adaptive gaussian kernel to obtain final interpolation data.
An adaptive Gaussian kernel based ISP color interpolation apparatus, comprising:
the RGB channel separation module is used for acquiring original data of an image to be processed, and R, G, B channel pixels are respectively obtained after RGB channel separation;
the self-adaptive Gaussian kernel generating module is used for traversing each pixel point to be interpolated on the G channel, respectively calculating correlation information between each pixel point to be interpolated and a neighborhood, and generating a self-adaptive Gaussian kernel by using the correlation information obtained by calculation, wherein the correlation information is used for controlling the rotation and the expansion of the Gaussian kernel;
the G channel interpolation module is used for interpolating pixel points to be interpolated according to the generated self-adaptive Gaussian core to obtain an interpolated G channel;
and the RGB channel fusion module is used for respectively interpolating R, B channels according to the interpolated G channel to obtain an interpolated R, B channel, and fusing R, G, B channels to obtain interpolated RGB data.
An adaptive gaussian kernel based ISP color interpolation apparatus comprising a processor and a memory, wherein the memory is used for storing a computer program, and the processor is used for executing the computer program, wherein the processor is used for executing the computer program to execute the method.
Compared with the prior art, the invention has the advantages that:
1. the invention generates the self-adaptive Gaussian kernel based on the correlation information between the pixel point and the neighborhood in the process of interpolating the G channel pixel, controls the rotation and the expansion of the Gaussian kernel by using the correlation information between the pixel point and the neighborhood, interpolates each pixel point of the G channel according to the generated self-adaptive Gaussian kernel, since the adaptive gaussian kernel controls rotation and scaling based on correlation information between pixel points and neighborhoods, therefore, the convolution by using the self-adaptive Gaussian kernel can generate different responses to the self-adaptation of the edge and non-edge positions on the image, the direction and the intensity of the edge area and the non-edge area can be sensed after interpolation, so that the edge area in the image can be reserved, the non-edge area can be smoothed, meanwhile, the distortion problem can be improved at the source generated by the distortion phenomenon, and the original structure of the image is reserved.
2. The invention further uses the covariance matrix as a matrix for controlling the rotation and the expansion of the Gaussian kernel, and can reflect the strength and the direction of the edge based on the covariance matrix, so that the energy of the Gaussian kernel is concentrated on the region of the edge of the image, thereby being capable of preserving the edge region in a self-adaptive manner, avoiding performing exponential calculation with natural logarithm as the base, accelerating the processing speed, and filtering the image while preserving the edge information.
3. The invention can greatly reduce the calculated amount of color interpolation and accelerate the interpolation processing speed by rationalizing the self-adaptive Gaussian kernel function on the basis of interpolation based on the self-adaptive Gaussian kernel function, is convenient to be applied to an FPGA platform based on a physicochemical processing mode, can effectively improve the processing efficiency and simultaneously reduces the consumption of resources.
4. The invention further carries out interpolation supplement on the sampling frequency of the R channel and the B channel, can reduce the false color effect, and simultaneously carries out bilinear interpolation on the R, B channel in a proportional mode based on the G channel, so that the information exchange among the three color channels is increased in the interpolation process of the R channel and the B channel, and the influence caused by different sampling frequencies on the RGB three channels can be counteracted.
Drawings
Fig. 1 is a schematic flow chart of an implementation process of the ISP color interpolation method based on the adaptive gaussian kernel in this embodiment.
FIG. 2 is a schematic diagram of a principle of using a Gaussian kernel and an adaptive Gaussian kernel.
Fig. 3 is a detailed flow chart of color interpolation based on adaptive gaussian core in an embodiment of the present invention.
Fig. 4 is a schematic diagram of a specific pipeline control for implementing color interpolation based on FPGA in a specific application embodiment of the present invention.
Detailed Description
The invention is further described below with reference to the drawings and specific preferred embodiments of the description, without thereby limiting the scope of protection of the invention.
As shown in fig. 1, the steps of the ISP color interpolation method based on the adaptive gaussian kernel in this embodiment include:
s1, RGB channel separation: acquiring original data of an image to be processed, and respectively obtaining R, G, B channel pixels after RGB channel separation;
s2, self-adaptive Gaussian kernel generation: traversing each pixel point to be interpolated on the G channel, respectively calculating correlation information between each pixel point to be interpolated and a neighborhood, and generating a self-adaptive Gaussian kernel by using the calculated correlation information, wherein the rotation and the expansion of the Gaussian kernel are controlled by using the correlation information;
s3.G channel interpolation: interpolating each pixel point to be interpolated according to the generated self-adaptive Gaussian core to obtain an interpolated G channel;
s4, RGB channel fusion: and respectively interpolating R, B channels according to the interpolated G channels to obtain R, B channels after interpolation, and fusing R, G, B channels after interpolation to obtain RGB data after interpolation.
The gaussian kernel is generally used for up-sampling and filtering the image, and the energy distribution of a typical gaussian kernel is shown in fig. 2(a), where the origin of the coordinate system represents the center of the neighborhood of the gaussian kernel, and the closer to the center, the higher the energy is, i.e., the higher the weight of the pixel there is. Edge information can be lost by directly adopting a Gaussian kernel for upsampling and filtering, the sharpening degree of an image is reduced, and when the edge of the image is processed, the edge information can be retained while the upsampling is carried out on the image by further improving the weight value at the edge. Assuming that the shape of the edge is as shown in fig. 2(b), where the dots are represented as target pixels, to achieve upsampling of the image while preserving the edge information, it is necessary to construct a gaussian kernel function as shown in fig. 2(c) under an energy distribution diagram.
In the embodiment, the above characteristics of the gaussian kernel function are considered, in the process of interpolating the G channel pixels, an adaptive gaussian kernel is generated based on the correlation information between the pixel points and the neighborhood, the rotation and expansion of the gaussian kernel are controlled by using the correlation information between the pixel points and the neighborhood, then each pixel point of the G channel is interpolated according to the generated adaptive gaussian kernel, since the adaptive gaussian kernel controls the rotation and expansion based on the correlation information between the pixel points and the neighborhood, the convolution using the adaptive gaussian kernel can adaptively generate different responses to each position of the edge and the non-edge on the image, the direction and the intensity of the edge region and the non-edge region can be perceived after interpolation, so that the edge region in the image can be reserved and the non-edge region can be smoothed, and the distortion problem can be improved at the source of the distortion phenomenon, the original structure of the image is preserved.
In this embodiment, the correlation information is specifically a covariance matrix in a neighborhood of a pixel to be interpolated, and the specific step of generating the adaptive gaussian kernel in step S2 includes: and calculating a covariance matrix in the neighborhood of each pixel point to be interpolated, and generating a corresponding self-adaptive Gaussian kernel by using each covariance matrix as a matrix for controlling rotation and expansion of the Gaussian kernel. Based on the covariance matrix, the correlation between the pixel points and the neighborhood pixel points can be effectively represented, so that the covariance matrix is used as a matrix for controlling the rotation and the expansion of the Gaussian kernel, the generated self-adaptive Gaussian kernel can accurately and self-adaptively sense the strength and the direction of the edge area, and the edge area can be accurately reserved.
A typical gaussian kernel function calculation formula is specifically:
x and y are respectively an abscissa and an ordinate of a pixel point in the image, h is a standard deviation of a Gaussian kernel, the concentration degree of the weight of the Gaussian kernel can be controlled by adjusting the standard deviation h, the smaller the standard deviation is, the larger the weight of the central part of pixels is, and otherwise, the smaller the weight is.
Writing the gaussian kernel function into a matrix form is:
wherein
To control the matrix of the gaussian kernel rotation and scaling, with this control matrix, rotation and scaling can only be achieved at specific locations.
In this embodiment, the covariance matrix is specifically obtained by calculating gradients of the pixel points to be interpolated in the x and y directions, that is, the covariance matrix is obtained by calculating the gradients of the pixel points to be interpolated in the x and y directions
As shown in fig. 2(d), the calculation formula of the adaptive gaussian kernel is:
wherein K (x, y) is an adaptive Gaussian kernel of the pixel point (x, y), C(x,y)Is the covariance matrix in the neighborhood of the pixel point (x, y).
Covariance matrix C(x,y)The intensity and direction of the edge can be reflected, and the embodiment uses the covariance matrix C(x,y)The method has the advantages that the edge of the image is responded, the rotation and the expansion of the Gaussian kernel are controlled, the energy of the Gaussian kernel can be concentrated on the area of the edge of the image, namely, the self-adaptive high weight is adopted in the area of the edge of the image, and therefore the edge area can be reserved in a self-adaptive mode.
It is understood that, in addition to the adaptive gaussian kernel shown in the above formula (1), other adaptive gaussian kernels may be used according to actual requirements.
The traditional method needs to perform index calculation based on natural logarithm based on Gaussian kernel filtering, has low calculation efficiency, is difficult to meet the real-time requirement of a camera carried on an unmanned vehicle, is insensitive to edge information in an image when being directly applied to color interpolation, and has fuzzy interpolation results globally. In the embodiment, the covariance matrix in the pixel neighborhood is calculated, the adaptive gaussian kernel is generated based on the covariance matrix for interpolation, and natural logarithm-based exponential calculation can be avoided, so that the processing speed can be increased, and the image is filtered while edge information is kept.
In step S2, the method further includes performing physicochemical calculation on the generated adaptive gaussian kernel to obtain a final physicochemical adaptive gaussian kernel output. The rational function is used for representing the original function, and a large amount of calculation still needs to be executed when the self-adaptive Gaussian kernel is directly used for interpolation.
In step S3, in this embodiment, a weighted average is specifically performed on the neighborhood of each pixel to be interpolated by using the value of the adaptive gaussian kernel, so as to obtain an interpolation value corresponding to the position of each pixel to be interpolated. Of course, other statistical calculation methods can be used to obtain the final interpolation according to the actual requirement, such as weighting
The step S4 of the present embodiment includes: and respectively calculating the ratio of the R, B channel to the G channel, and performing bilinear interpolation on the R, B channel according to the calculated ratio to obtain a R, B channel after interpolation. The traditional ISP color interpolation algorithm based on bilinear interpolation needs to be limited by the difference of sampling frequencies of different color channels, the photosensitive elements of the embedded image sensor are usually arranged in an RGGB or BGGR mode, and the sampling frequency on the G channel is twice that of the R channel and the B channel, so that the color interpolation algorithm based on bilinear interpolation can cause local spatial misalignment of the RGB channels. In the embodiment, the pseudo color effect can be reduced by performing interpolation supplement on the sampling frequencies of the R channel and the B channel, and meanwhile, the R, B channel is subjected to bilinear interpolation in a proportional mode based on the G channel, so that information exchange among three color channels is increased in the interpolation process of the R channel and the B channel, and the influence caused by different sampling frequencies on the RGB three channels can be offset.
As shown in fig. 3, the process of implementing color interpolation in the embodiment of the present invention includes:
step 1: after preprocessing such as dead pixel detection and the like is carried out on input original Bayer grid data, separating an RGGB channel of an original Bayer grid to obtain an R, G, B channel;
step 2: performing pixel traversal on the separated G channel, and calculating a covariance matrix in the neighborhood of the specified range (specifically, 5 x 5) of each pixel point to be interpolated;
and step 3: calculating a physicochemical adaptive Gaussian kernel according to the covariance matrix of each pixel neighborhood;
and 4, step 4: carrying out weighted average on the neighborhood by using values in the physicochemical adaptive Gaussian kernel to obtain the interpolation of each pixel point position;
and 5: calculating R, B a ratio of channel to G channel, wherein only a portion of R, B channel with value is calculated;
step 6: carrying out bilinear interpolation on the R, B channel by combining the ratio of the R, B channel to the G channel to obtain an interpolation result of the R, B channel;
and 7: and fusing interpolation results of all channels and outputting RGB data.
In this embodiment, the FPGA is used to control and execute the steps of the color interpolation method according to a pipeline control manner, so that the whole processing process uses a pipeline processing manner, and the interpolation processing speed and the data throughput are further improved by using the advantage of parallel processing of the FPGA.
In this embodiment, when the FPGA is used according to a pipeline control method, G channel interpolation and RB channel interpolation are performed in sequence, where the G channel interpolation is divided into multiple stages of pipelines for gradient calculation, covariance matrix calculation, weight calculation, and G channel interpolation calculation, a gradient is calculated by the gradient calculation stage pipeline, a covariance matrix is calculated by the covariance matrix calculation stage pipeline based on the calculated gradient, a weight required for interpolation is calculated by the weight calculation stage pipeline, and then calculation of a physicochemical adaptive gaussian kernel is completed by the G channel interpolation stage pipeline to obtain a final G channel interpolation result. In the FPGA flow control process, the G channel interpolation is calculated based on a physicochemical adaptive Gaussian kernel function mode, so that the interpolation processing can be efficiently executed on an FPGA platform, and meanwhile, the resource consumption required in the interpolation process can be reduced.
In a specific application embodiment, the steps of the color interpolation method are divided into multiple stages of pipelines, and the steps that can be executed in parallel are configured to be executed in parallel, as shown in fig. 4, the division of each stage of pipelines specifically includes:
(1) g, preprocessing interpolation data: and generating data blocks required by gradient calculation and preprocessing the image edges. Specifically, a 7 × 7 data block required by gradient calculation is generated by using a tap method, the edge of an image is processed, and the processed data is subjected to clock domain crossing processing;
(2) gradient calculation: and calculating gradient values of the non-green pixel points of the pixel block in the x direction and the y direction by using a plurality of parallelism degrees. Specifically, 7 parallelism degrees are used, gradient values of 13/12 non-green pixel points in the 5 x 5 pixel blocks in the x direction and the y direction are calculated in two times, and the calculation can be specifically configured according to actual requirements;
(3) and (3) covariance matrix calculation: calculating a product of gradients using a plurality of degrees of parallelism, calculating a square of an x-gradient, a square of a y-gradient, and a product of the two gradients, respectively; and grouping the solved products and then accumulating. Specifically, 7 parallelism degrees are used for calculating the product of the gradients, each parallelism degree comprises three integer multipliers, and the product of the square of the x gradient, the square of the y gradient and two gradients is calculated respectively; dividing the 7 solved products into two groups, wherein one group is 4-number addition, the other group is 3-number addition, and finally accumulating the two sums;
(4) calculating a weight value: and grouping to calculate the weight. The weight calculation amount is changed into a practical half (7/6) due to the symmetry of the weights, wherein the weight is equal to zero at the center of a 5 x 5 pixel block, so that the calculation is not performed; the calculation is divided into two groups, and three weights are calculated each time;
(5) and G channel interpolation calculation: the method comprises the following steps of performing interpolation calculation, wherein the whole calculation uses a flow mode, firstly, the weight and the product of the weight and a green pixel point are respectively summed, then, division operation is performed, namely, the calculation of a physicochemical self-adaptive Gaussian kernel is completed, and interpolation data are calculated;
(6) g, synchronously outputting interpolation: outputting two paths of data, wherein one path of data is original image data which is not interpolated, the other path of data is image data which is interpolated by a G channel, and the two paths of data are synchronously output after being processed across clock domains;
(7) preprocessing RB channel interpolation data: and generating a data block required by interpolation, and preprocessing the edge of the image. Specifically, a 3 x 3 data block required by interpolation is generated by using a tap method, the edge of an image is processed, and the processed data is processed in a clock domain crossing manner;
(8) RB interpolation: calculating interpolation data of R and B channels by using the original data and the interpolated G channel data, and calculating the interpolation of R and B channels by using the same logic resource due to the alternative occurrence of RB;
(9) RB interpolation synchronization output: and synchronously outputting RGB component data after performing clock domain crossing processing on the RB data after interpolation and the G channel data after interpolation.
Through the flow division, the parallel advantage of the FPGA platform can be fully exerted, the flows capable of being processed in parallel are performed in parallel as far as possible, the color interpolation can be efficiently realized, and the interpolation effect is ensured.
It can be understood that, besides the FPGA executing the pipeline division manner, other pipeline division manners and parallel configuration manners may be adopted according to actual requirements.
The present embodiment further provides an ISP color interpolation apparatus based on an adaptive gaussian kernel, including:
the RGB channel separation module is used for acquiring original data of an image to be processed, and R, G, B channel pixels are respectively obtained after RGB channel separation;
the self-adaptive Gaussian kernel generating module is used for traversing each pixel point to be interpolated on the G channel, respectively calculating correlation information between each pixel point to be interpolated and a neighborhood, and generating a self-adaptive Gaussian kernel by using the correlation information obtained by calculation, wherein the correlation information is used for controlling the rotation and the expansion of the Gaussian kernel;
the G channel interpolation module is used for interpolating each pixel point to be interpolated according to the generated self-adaptive Gaussian core to obtain an interpolated G channel;
and the RGB channel fusion module is used for respectively interpolating R, B channels according to the interpolated G channel to obtain an interpolated R, B channel, and fusing R, G, B channels to obtain interpolated RGB data.
In this embodiment, the ISP color interpolation apparatus based on the adaptive gaussian kernel corresponds to the ISP color interpolation method based on the adaptive gaussian kernel one to one, and details are not repeated here.
In another embodiment, the ISP color interpolation apparatus based on the adaptive gaussian kernel may further be: the method comprises a processor and a memory, wherein the memory is used for storing computer programs, the processor is used for executing the computer programs, and the processor is used for executing the computer programs so as to execute the ISP color interpolation method based on the adaptive Gaussian kernel.
The foregoing is considered as illustrative of the preferred embodiments of the invention and is not to be construed as limiting the invention in any way. Although the present invention has been described with reference to the preferred embodiments, it is not intended to be limited thereto. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical spirit of the present invention should fall within the protection scope of the technical scheme of the present invention, unless the technical spirit of the present invention departs from the content of the technical scheme of the present invention.
Claims (10)
1. An ISP color interpolation method based on an adaptive Gaussian kernel is characterized by comprising the following steps:
s1, RGB channel separation: acquiring original data of an image to be processed, and respectively obtaining R, G, B channel pixels after RGB channel separation;
s2, self-adaptive Gaussian kernel generation: traversing each pixel point to be interpolated on the G channel, respectively calculating correlation information between each pixel point to be interpolated and a neighborhood, and generating a self-adaptive Gaussian kernel by using the correlation information obtained by calculation, wherein the correlation information is used for controlling the rotation and the expansion of the Gaussian kernel;
s3.G channel interpolation: interpolating each pixel point to be interpolated according to the generated self-adaptive Gaussian core to obtain an interpolated G channel;
s4, RGB channel fusion: and respectively interpolating R, B channels according to the interpolated G channels to obtain R, B channels after interpolation, and fusing R, G, B channels after interpolation to obtain RGB data after interpolation.
2. The ISP color interpolation method based on the adaptive gaussian kernel as claimed in claim 1, wherein the correlation information is a covariance matrix in a neighborhood of a pixel to be interpolated, and the specific step of generating the adaptive gaussian kernel in step S2 includes: and calculating a covariance matrix in the neighborhood of each pixel point to be interpolated, and generating a corresponding self-adaptive Gaussian kernel by using each covariance matrix as a matrix for controlling rotation and expansion of the Gaussian kernel.
3. The adaptive gaussian kernel-based ISP color interpolation method of claim 2, wherein: the covariance matrix is obtained by calculating gradients of pixel points to be interpolated in x and y directions.
4. The ISP color interpolation method based on the adaptive gaussian kernel as claimed in claim 1, wherein the step S2 further comprises performing physicochemical calculation on the generated adaptive gaussian kernel to obtain a final physicochemical adaptive gaussian kernel output.
5. The ISP color interpolation method based on the adaptive Gaussian kernel as claimed in any one of claims 1 to 4, wherein in the step S3, the adaptive Gaussian kernel value is specifically used to perform weighted average on the neighborhood of each pixel to be interpolated, so as to obtain the interpolation corresponding to the position of each pixel to be interpolated.
6. The ISP color interpolation method based on the adaptive Gaussian kernel as claimed in any one of claims 1 to 4, wherein the step S4 comprises the following steps: and respectively calculating the ratio between the R, B channel and the G channel, and performing bilinear interpolation on the R, B channel according to the calculated ratio to obtain a R, B channel after interpolation.
7. The ISP color interpolation method based on the adaptive Gaussian kernel according to any one of claims 1 to 4, characterized in that each step in the color interpolation method is controlled and executed by adopting an FPGA according to a pipeline control mode.
8. The ISP color interpolation method based on the adaptive Gaussian kernel according to claim 7, wherein the G channel interpolation and the RB channel interpolation are sequentially performed by adopting the FPGA according to a flow control mode, wherein the G channel interpolation is divided into a plurality of stages of flows of gradient calculation, covariance matrix calculation, weight calculation and G channel interpolation calculation, the gradient is calculated by the gradient calculation stage flow, the covariance matrix is calculated by the covariance matrix calculation stage flow based on the calculated gradient, the weight required by the interpolation is calculated by the weight calculation stage flow, and the calculation of the physicochemical adaptive Gaussian kernel is completed by the G channel interpolation stage flow to obtain a final G channel interpolation result.
9. An adaptive gaussian kernel-based ISP color interpolation apparatus, comprising:
the RGB channel separation module is used for acquiring original data of an image to be processed, and R, G, B channel pixels are respectively obtained after RGB channel separation;
the self-adaptive Gaussian kernel generating module is used for traversing each pixel point to be interpolated on the G channel, respectively calculating correlation information between each pixel point to be interpolated and a neighborhood, and generating a self-adaptive Gaussian kernel by using the correlation information obtained by calculation, wherein the correlation information is used for controlling the rotation and the expansion of the Gaussian kernel;
the G channel interpolation module is used for interpolating pixel points to be interpolated according to the generated self-adaptive Gaussian core to obtain an interpolated G channel;
and the RGB channel fusion module is used for respectively interpolating R, B channels according to the interpolated G channel to obtain an interpolated R, B channel, and fusing R, G, B channels to obtain interpolated RGB data.
10. An adaptive gaussian kernel based ISP color interpolation apparatus, comprising a processor and a memory, wherein the memory is used for storing a computer program, and the processor is used for executing the computer program, wherein the processor is used for executing the computer program to execute the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011096033.2A CN112233019B (en) | 2020-10-14 | 2020-10-14 | ISP color interpolation method and device based on self-adaptive Gaussian kernel |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011096033.2A CN112233019B (en) | 2020-10-14 | 2020-10-14 | ISP color interpolation method and device based on self-adaptive Gaussian kernel |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112233019A true CN112233019A (en) | 2021-01-15 |
| CN112233019B CN112233019B (en) | 2023-06-30 |
Family
ID=74112785
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011096033.2A Active CN112233019B (en) | 2020-10-14 | 2020-10-14 | ISP color interpolation method and device based on self-adaptive Gaussian kernel |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112233019B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115063460A (en) * | 2021-12-24 | 2022-09-16 | 山东建筑大学 | High-precision self-adaptive homonymous pixel interpolation and optimization method |
| CN115689887A (en) * | 2022-10-28 | 2023-02-03 | 辉羲智能科技(上海)有限公司 | Automatic driving RGBIR image resampling method, system, terminal and medium |
| CN115689895A (en) * | 2023-01-01 | 2023-02-03 | 武汉图科智能科技有限公司 | Up-sampling hardware processing system based on Lanczos interpolation |
| CN116912096A (en) * | 2023-09-14 | 2023-10-20 | 惠州市耀盈精密技术有限公司 | IMOLD system-based precision mold template image generation method |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7412429B1 (en) * | 2007-11-15 | 2008-08-12 | International Business Machines Corporation | Method for data classification by kernel density shape interpolation of clusters |
| EP2124187A1 (en) * | 2008-05-22 | 2009-11-25 | Telefonaktiebolaget LM Ericsson (PUBL) | Apparatus and method for demosaicing |
| CN103686111A (en) * | 2013-12-31 | 2014-03-26 | 上海富瀚微电子有限公司 | Method and device for correcting color based on RGBIR (red, green and blue, infra red) image sensor |
| CN103995860A (en) * | 2014-05-16 | 2014-08-20 | 中国科学院重庆绿色智能技术研究院 | Image retrieval system and method based on color feature extraction |
| CN104794691A (en) * | 2015-04-07 | 2015-07-22 | 浙江大学 | Definition reconstruction method of single out-of-focus image using generalized Gaussian model |
| EP3051484A2 (en) * | 2015-01-30 | 2016-08-03 | BOE Technology Group Co., Ltd. | Image interpolation method and image interpolation apparatus |
| CN106910168A (en) * | 2017-01-09 | 2017-06-30 | 中国科学院自动化研究所 | Parallel image color enhancement method and apparatus |
| CN107403435A (en) * | 2017-07-28 | 2017-11-28 | 江苏大学 | A kind of Color Image Edge extracting method based on RGB color |
| CN110706174A (en) * | 2019-09-27 | 2020-01-17 | 集美大学 | Image enhancement method, terminal equipment and storage medium |
| JP2020155907A (en) * | 2019-03-20 | 2020-09-24 | 株式会社朋栄 | Automatic color correction processing method and automatic color correction processing device |
-
2020
- 2020-10-14 CN CN202011096033.2A patent/CN112233019B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7412429B1 (en) * | 2007-11-15 | 2008-08-12 | International Business Machines Corporation | Method for data classification by kernel density shape interpolation of clusters |
| EP2124187A1 (en) * | 2008-05-22 | 2009-11-25 | Telefonaktiebolaget LM Ericsson (PUBL) | Apparatus and method for demosaicing |
| CN103686111A (en) * | 2013-12-31 | 2014-03-26 | 上海富瀚微电子有限公司 | Method and device for correcting color based on RGBIR (red, green and blue, infra red) image sensor |
| CN103995860A (en) * | 2014-05-16 | 2014-08-20 | 中国科学院重庆绿色智能技术研究院 | Image retrieval system and method based on color feature extraction |
| EP3051484A2 (en) * | 2015-01-30 | 2016-08-03 | BOE Technology Group Co., Ltd. | Image interpolation method and image interpolation apparatus |
| CN104794691A (en) * | 2015-04-07 | 2015-07-22 | 浙江大学 | Definition reconstruction method of single out-of-focus image using generalized Gaussian model |
| CN106910168A (en) * | 2017-01-09 | 2017-06-30 | 中国科学院自动化研究所 | Parallel image color enhancement method and apparatus |
| CN107403435A (en) * | 2017-07-28 | 2017-11-28 | 江苏大学 | A kind of Color Image Edge extracting method based on RGB color |
| JP2020155907A (en) * | 2019-03-20 | 2020-09-24 | 株式会社朋栄 | Automatic color correction processing method and automatic color correction processing device |
| CN110706174A (en) * | 2019-09-27 | 2020-01-17 | 集美大学 | Image enhancement method, terminal equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 赵景秀;王菁;赵昭;: "基于RGB空间剖分的彩色图像边缘检测", 光电子技术 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115063460A (en) * | 2021-12-24 | 2022-09-16 | 山东建筑大学 | High-precision self-adaptive homonymous pixel interpolation and optimization method |
| CN115689887A (en) * | 2022-10-28 | 2023-02-03 | 辉羲智能科技(上海)有限公司 | Automatic driving RGBIR image resampling method, system, terminal and medium |
| CN115689895A (en) * | 2023-01-01 | 2023-02-03 | 武汉图科智能科技有限公司 | Up-sampling hardware processing system based on Lanczos interpolation |
| CN115689895B (en) * | 2023-01-01 | 2023-04-07 | 武汉图科智能科技有限公司 | Up-sampling hardware processing system based on Lanczos interpolation |
| CN116912096A (en) * | 2023-09-14 | 2023-10-20 | 惠州市耀盈精密技术有限公司 | IMOLD system-based precision mold template image generation method |
| CN116912096B (en) * | 2023-09-14 | 2024-01-09 | 惠州市耀盈精密技术有限公司 | IMOLD system-based precision mold template image generation method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112233019B (en) | 2023-06-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112233019A (en) | ISP color interpolation method and device based on self-adaptive Gaussian kernel | |
| CN108376392B (en) | Image motion blur removing method based on convolutional neural network | |
| US20220036523A1 (en) | Image processor | |
| WO2017044292A1 (en) | Automatic compensation of lens flare | |
| CN113034391B (en) | Multi-mode fusion underwater image enhancement method, system and application | |
| CN111340732B (en) | Low-illumination video image enhancement method and device | |
| CN105046681A (en) | Image salient region detecting method based on SoC | |
| CN111008936B (en) | Multispectral image panchromatic sharpening method | |
| WO2022061879A1 (en) | Image processing method, apparatus and system, and computer-readable storage medium | |
| CN103886565A (en) | Nighttime color image enhancement method based on purpose optimization and histogram equalization | |
| Mei et al. | Higher-resolution network for image demosaicing and enhancing | |
| KR20190059157A (en) | Method and Apparatus for Improving Image Quality | |
| CN115564692B (en) | Full color-multispectral-hyperspectral integrated fusion method considering breadth difference | |
| Zhong et al. | Deep attentional guided image filtering | |
| CN103907134A (en) | Image processing device, image processing method, image-capturing device, computer program, and recording medium | |
| CN101639932A (en) | Method and system for enhancing digital image resolution | |
| US8452090B1 (en) | Bayer reconstruction of images using a GPU | |
| CN115004220A (en) | Neural network for raw low-light image enhancement | |
| Zhao et al. | Saliency map-aided generative adversarial network for raw to rgb mapping | |
| CN104038746A (en) | Data interpolation method for BAYER format image | |
| CN115170921A (en) | Binocular stereo matching method based on bilateral grid learning and edge loss | |
| CN114565764A (en) | Port panorama sensing system based on ship instance segmentation | |
| US9779470B2 (en) | Multi-line image processing with parallel processing units | |
| JP7512150B2 (en) | Information processing device, information processing method, and program | |
| Lin et al. | Searching lightweight neural network for image signal processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information |
Inventor after: Guo Wang Inventor after: Wu Qingqing Inventor after: Hu Tingbo Inventor after: An Xiangjing Inventor after: Zhang Mengxuan Inventor before: Guo Wang Inventor before: Hu Tingbo Inventor before: An Xiangjing Inventor before: Zhang Mengxuan |
|
| CB03 | Change of inventor or designer information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |