CN110852953A - Image interpolation method and device, storage medium, image signal processor and terminal - Google Patents
Image interpolation method and device, storage medium, image signal processor and terminal Download PDFInfo
- Publication number
- CN110852953A CN110852953A CN201911121851.0A CN201911121851A CN110852953A CN 110852953 A CN110852953 A CN 110852953A CN 201911121851 A CN201911121851 A CN 201911121851A CN 110852953 A CN110852953 A CN 110852953A
- Authority
- CN
- China
- Prior art keywords
- plane
- green
- full
- red
- initial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4007—Interpolation-based scaling, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4023—Decimation- or insertion-based scaling, e.g. pixel or line decimation
Abstract
An image interpolation method and device, a storage medium, an image signal processor and a terminal are provided, wherein the image interpolation method comprises the following steps: moving a sliding window in the Bayer image to be interpolated, and separating pixels in the sliding window according to a color channel; after moving the sliding window each time, carrying out interpolation on the original green plane; calculating an initial red-green middle plane and an initial blue-green middle plane according to the green full plane, the original red plane and the original blue plane; interpolating an initial red-green middle plane and an initial blue-green middle plane according to the green full plane; calculating a red full plane and a blue full plane at least according to the initial red-green middle full plane, the initial blue-green middle full plane and the green full plane; and synthesizing a color image according to the green full plane, the red full plane and the blue full plane until the Bayer image to be interpolated is traversed. The technical scheme of the invention can improve the estimation accuracy of the Bell interpolation and improve the image quality.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an image interpolation method and apparatus, a storage medium, an image signal processor, and a terminal.
Background
For an image, one pixel has a mixed color composed of three colors of RGB, while a bayer (bayer) image has only one color, or R, G, or B, for one pixel. Therefore, the color of the pixel points around the pixel point in the bayer graph needs to be interpolated (filled) to obtain another two colors.
For conventional bayer images, one typically needs to estimate the missing pixel values on the RGB channel plane in various ways. It is common practice to have linear interpolation, adjacent interpolation, etc. However, in order to obtain a better interpolation effect, more interpolation methods are proposed, and the conventional interpolation target data is mainly divided into two types: interpolation algorithms based on the mid-plane and on the color ratio plane. Since the intermediate plane has a small transformation with the change of the image content, the estimation of the missing pixels is performed on the plane with small fluctuation, and the error introduced by the estimation is also greatly reduced. In addition, there have recently been some approaches based on residual estimation after multiple estimations to improve the accuracy of the missing pixel estimation. Most bayer (bayer) interpolations currently obtain interpolation results through a single estimation, while some can obtain interpolation results through an iterative calculation mode.
However, in the existing interpolation algorithm, only the neighborhood pixels are used for complementing the default color, and the image quality after interpolation is low.
Disclosure of Invention
The invention solves the technical problem of how to improve the estimation accuracy of the Bell interpolation and improve the image quality.
In order to solve the above technical problem, an embodiment of the present invention provides an image interpolation method, where the image interpolation method includes: obtaining a Bayer image to be interpolated; moving a sliding window in the Bayer image to be interpolated, and separating pixels in the sliding window according to color channels to obtain each original color plane; after moving the sliding window each time, performing interpolation on an original green plane to obtain a green full plane, wherein each pixel in the green full plane has a green component value; after the sliding window is moved each time, calculating an initial red-green middle plane and an initial blue-green middle plane according to the green full plane, the initial red plane and the initial blue-green plane; after the sliding window is moved each time, interpolating the initial red-green middle plane and the initial blue-green middle plane according to the green full plane to obtain an initial red-green middle full plane and an initial blue-green middle full plane, wherein each pixel in the initial red-green middle full plane and the initial blue-green middle full plane has a color component difference value; after each movement of the sliding window, calculating a red full plane and a blue full plane according to at least the initial red-green intermediate full plane, the initial blue-green intermediate full plane and the green full plane, each pixel in the red full plane having a red component value and each pixel in the blue full plane having a blue component value; and after the sliding window is moved each time, synthesizing a color image according to the green full plane, the red full plane and the blue full plane until the Bayer image to be interpolated is traversed.
Optionally, the calculating the red full plane and the blue full plane according to at least the initial red-green intermediate full plane, the initial blue-green intermediate full plane, and the green full plane includes: calculating an initial red full plane and an initial blue full plane according to the initial red-green middle full plane, the initial blue-green middle full plane and the green full plane; calculating a luminance plane using the green full plane, the initial red full plane, and the initial blue full plane; guiding and filtering the initial red-green intermediate full plane and the initial blue-green intermediate full plane by utilizing the brightness information of the brightness plane to obtain a final red-green intermediate full plane and a final blue-green intermediate full plane; and calculating the red full plane and the blue full plane according to the final red-green intermediate full plane, the final blue-green intermediate full plane and the green full plane.
Optionally, the calculating the red full plane and the blue full plane according to at least the initial red-green intermediate full plane, the initial blue-green intermediate full plane, and the green full plane includes: calculating the sum of the initial red-green middle full plane and the green full plane to obtain the red full plane; calculating a sum of the initial cyan intermediate global plane and the green global plane to obtain the blue global plane.
Optionally, the interpolating the initial red-green intermediate plane and the initial blue-green intermediate plane according to the green full plane includes: and respectively guiding and filtering the initial red-green middle plane and the initial blue-green middle plane according to the green full plane.
Optionally, the interpolating on the original green plane includes: calculating the gradients of the original green plane in the horizontal direction and the vertical direction; and selecting the direction with the minimum gradient value to interpolate the default pixels of the original green plane so as to obtain the green full plane.
Optionally, each pixel in the color image has a red component value, a blue component value, and a green component value.
In order to solve the above technical problem, an embodiment of the present invention further discloses an image interpolation apparatus, including: the Bayer image acquisition module is used for acquiring a Bayer image to be interpolated; the color channel separation module is used for moving a sliding window in the Bayer image to be interpolated and separating pixels in the sliding window according to color channels to obtain each original color plane; the green full-plane interpolation module is used for interpolating on the original green plane after the sliding window is moved every time to obtain a green full plane, and each pixel in the green full plane has a green component value; the middle plane calculation module is used for calculating an initial red-green middle plane and an initial blue-green middle plane according to the green full plane, the original red plane and the original blue plane after the sliding window is moved each time; the intermediate full plane calculation module is used for interpolating the initial red and green intermediate plane and the initial blue and green intermediate plane according to the green full plane after the sliding window is moved every time so as to obtain an initial red and green intermediate full plane and an initial blue and green intermediate full plane, and each pixel in the initial red and green intermediate full plane and the initial blue and green intermediate full plane has a color component difference value; a red-blue full plane calculation module for calculating a red full plane and a blue full plane at least according to the initial red-green intermediate full plane, the initial blue-green intermediate full plane and the green full plane after moving the sliding window each time, wherein each pixel in the red full plane has a red component value, and each pixel in the blue full plane has a blue component value; and the color image calculation module is used for synthesizing a color image according to the green full plane, the red full plane and the blue full plane after the sliding window is moved each time until the Bayer image to be interpolated is traversed.
The embodiment of the invention also discloses a storage medium, wherein computer instructions are stored on the storage medium, and the steps of the image interpolation method are executed when the computer instructions are executed.
The embodiment of the invention also discloses an image signal processor, which comprises a memory and a processor, wherein the memory is stored with computer instructions capable of running on the processor, and the processor executes the steps of the image interpolation method when running the computer instructions.
The embodiment of the invention also discloses a terminal which comprises a memory and a processor, wherein the memory is stored with a computer instruction which can be operated on the processor, and the processor executes the steps of the image interpolation method when operating the computer instruction.
Compared with the prior art, the technical scheme of the embodiment of the invention has the following beneficial effects:
in the technical scheme of the invention, because the original green channel information in the Bayer image has better signal-to-noise ratio, the green full plane can be obtained by interpolation first. And calculating an initial red-green intermediate plane and an initial blue-green intermediate plane by utilizing the green component values of all the pixels in the green full plane and the original red plane and the original blue plane. The intermediate full plane can be obtained by performing interpolation and completion on the two intermediate planes, and then the red full plane and the blue full plane are obtained by calculation, so that the color image of each pixel with red, green and blue component values can be obtained. Because the green full plane is used as the reference information for calculating and complementing the middle plane, the final color image has better signal-to-noise ratio, and the image quality is improved.
Furthermore, the technical scheme of the invention can also calculate the brightness plane according to the preliminarily calculated green full plane, the initial red full plane and the initial blue full plane, and filter the middle full plane by utilizing the brightness information of the brightness plane, and because the brightness information, the signal-to-noise ratio and the content information are much better than those of a single red or blue channel, the information of the final red and green middle full plane and the final blue and green middle full plane is richer, the signal-to-noise ratio is better, and the image quality of the final color image is further improved.
Drawings
FIG. 1 is a flow chart of a method of image interpolation according to an embodiment of the present invention;
FIG. 2 is a schematic illustration of a Bayer image according to an embodiment of the invention;
FIG. 3 is a schematic diagram of an original red color plane according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an original blue plane according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of an original green plane according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of an initial red-green midplane of an embodiment of the invention;
FIG. 7 is a schematic view of an initial red-green intermediate global plane according to an embodiment of the present invention;
FIG. 8 is a flowchart of one embodiment of step S106 shown in FIG. 1;
FIG. 9 is a diagram illustrating an exemplary application scenario of the present invention;
fig. 10 is a schematic structural diagram of an image interpolation apparatus according to an embodiment of the present invention.
Detailed Description
As described in the background art, in the existing interpolation algorithm, only the neighborhood pixels are used to complement the default color, and the image quality after interpolation is low.
The inventor of the application finds that on one hand, noise exists in r, g and b channels of a bayer image originally, although in most ISP pipeline, denoising processing is carried out before bayer interpolation is carried out. But the bayer domain denoising processing is not suitable to be strong for the imaging effect of the whole ISP system. Even for denoising methods with better detail preservation, such as edge protection filtering, block similarity matching, and the like, too strong bayer denoising processing causes loss of image information, especially loss of details. For the entire ISP system, the information lost at the very front of the ISP is difficult to recover by the following enhancement modules. Therefore, even if a denoising processing module exists before the bayer interpolation, the noise cannot be completely removed. On the other hand, the interpolation method itself is a method of estimation by surrounding pixel information, and is not a true measurement value. The estimation method itself introduces errors. And such errors are not merely on a single channel. There are substantially mutually independent signals from pixel to pixel prior to bayer interpolation. But after the bayer interpolation, the error of the central pixel introduces error components of the adjacent surrounding pixels in addition to the error of the self-measured signal. The errors between different pixels are thus operationally correlated. Although most ispppieline performs denoising processing of brightness (luma) and chroma (chroma) on the ycbcr space at the back end. But the RGB space before ycbcr usually does some non-linear operation. Such as gamma, local tone mapping (local tone mapping), etc. This series of processing may exacerbate the amplification of image noise. Therefore, the defects related to noise and the like on the image can be better processed before the nonlinear area.
That is, most of the prior bayer interpolations only aim at estimation of missing pixels, but the estimated value deviation is rarely processed.
In the technical scheme of the invention, because the original green channel information in the Bayer image has better signal-to-noise ratio, the green full plane can be obtained by interpolation first. And calculating an initial red-green intermediate plane and an initial blue-green intermediate plane by utilizing the green component values of all the pixels in the green full plane and the original red plane and the original blue plane. The intermediate full plane can be obtained by performing interpolation and completion on the two intermediate planes, and then the red full plane and the blue full plane are obtained by calculation, so that the color image of each pixel with red, green and blue component values can be obtained. Because the green full plane is used as the reference information for calculating and complementing the middle plane, the final color image has better signal-to-noise ratio, and the image quality is improved.
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below.
Fig. 1 is a flowchart of an image interpolation method according to an embodiment of the present invention.
The image interpolation method of the embodiment of the present invention may be used on a terminal device side having an image processing capability, that is, the terminal device may execute each step of the method shown in fig. 1.
The image interpolation method shown in fig. 1 may include the steps of:
step S101: obtaining a Bayer image to be interpolated;
step S102: moving a sliding window in the Bayer image to be interpolated, and separating pixels in the sliding window according to color channels to obtain each original color plane;
step S103: after moving the sliding window each time, performing interpolation on an original green plane to obtain a green full plane, wherein each pixel in the green full plane has a green component value;
step S104: after the sliding window is moved each time, calculating an initial red-green middle plane and an initial blue-green middle plane according to the green full plane, the initial red plane and the initial blue-green plane;
step S105: after the sliding window is moved each time, interpolating the initial red-green middle plane and the initial blue-green middle plane according to the green full plane to obtain an initial red-green middle full plane and an initial blue-green middle full plane, wherein each pixel in the initial red-green middle full plane and the initial blue-green middle full plane has a color component difference value;
step S106: after each movement of the sliding window, calculating a red full plane and a blue full plane according to at least the initial red-green intermediate full plane, the initial blue-green intermediate full plane and the green full plane, each pixel in the red full plane having a red component value and each pixel in the blue full plane having a blue component value;
step S107: and after the sliding window is moved each time, synthesizing a color image according to the green full plane, the red full plane and the blue full plane until the Bayer image to be interpolated is traversed.
It should be noted that the sequence numbers of the steps in this embodiment do not represent a limitation on the execution sequence of the steps.
In the implementation of step S101, each pixel in the bayer image to be interpolated has only a single color component, and specifically, referring to fig. 2, each pixel has a color component of R, G (e.g., Gr or Gb) or B.
In steps S102 to S107, processing may be performed for pixels within the sliding window. The size of the sliding window may be M pixels × N pixels, where specific values of M and N may be set according to an actual application scenario, which is not limited in this embodiment of the present invention.
In a specific implementation of step S102, the pixels in the sliding window may be separated according to color channels to obtain original color planes, that is, an original red plane, an original green plane, and an original blue plane, which may be specifically referred to in fig. 3 to 5. And the size of each original color plane is the same as that of the Bayer image to be interpolated. Specifically, referring to fig. 3, in the original red plane, the pixels with colors only have red component values, and the remaining pixels (shown as white positions in the figure) are the pixels to be interpolated; referring to fig. 4, in the original blue plane, the pixel value with color only has a blue component value, and the remaining pixels (shown as white positions in the figure) are the pixels to be interpolated; referring to FIG. 5, in the original green plane, the pixel value with color only has the green component value, and the remaining pixels (shown as white positions in the figure) are the pixels to be interpolated.
In a specific implementation of step S103, the default pixels in the original green plane shown in fig. 5 may be interpolated to obtain a green global plane, where each pixel in the green global plane has a green component value.
Further, in the specific implementation of step S104, an initial red-green intermediate plane and an initial blue-green intermediate plane may be calculated according to the green full plane, the original red plane and the original blue plane. Specifically, the difference between the original red plane and the green full plane can be utilized to obtain an initial red-green middle plane; and (4) obtaining an initial blue-green middle plane by utilizing the difference between the original blue plane and the green full plane. Specifically, referring to FIG. 6, the pixel values of the pixels with colors in the initial red-green middle plane are R-G, for example, the pixel values of the pixels at the positions (1,1) are R-G. That is, R-G is used to represent the difference between the red component and the green component, and B-G is used to represent the difference between the blue component and the green component.
In a specific embodiment, the original red plane may be divided by the green full plane to obtain an initial red-green intermediate plane; the original blue plane is divided by the green full plane to obtain the initial cyan mid-plane. For example, the pixel value of a pixel having a color in the initial red-green intermediate plane is R/G; the pixel values of the pixels having colors in the initial cyan mid-plane are B/G. That is, R/G is used to represent the difference between the red component and the green component, and B/G is used to represent the difference between the blue component and the green component.
In another embodiment, the original red plane and the green full plane may be summed to obtain an initial red-green intermediate plane; and (4) summing the original blue plane and the green full plane to obtain an initial blue-green middle plane. For example, the pixel value of a pixel with a color in the initial red-green intermediate plane is R + G; the pixel value of the pixel with the color in the initial cyan midplane is B + G. That is, R + G is used to represent the difference between the red component and the green component, and B + G is used to represent the difference between the blue component and the green component.
It should be noted that the initial red-green middle plane and the initial blue-green middle plane may also be calculated by any other implementable manner, which is not limited in the embodiment of the present invention.
Since the original red plane and the original blue plane have default pixels (e.g., pixels lacking a red component value and pixels lacking a blue component value) in them, the original red-green intermediate plane and the original blue-green intermediate plane also have default pixels.
In a specific implementation of step S105, the initial red-green intermediate plane and the initial blue-green intermediate plane may be interpolated according to information such as image details provided by the green global plane, for example, details such as flatness, edge, texture, etc., to fill default pixels in the initial red-green intermediate plane and the initial blue-green intermediate plane. Each pixel in the initial red-green intermediate global plane (as shown in fig. 7) and the initial blue-green intermediate global plane has a color difference value. Specifically, the pixel value of each pixel in the initial red-green middle full plane is R-G; the pixel values of each pixel in the initial cyan intermediate global plane are B-G.
Further in a specific implementation of step S106, the red full plane and the blue full plane are calculated at least according to the initial red-green intermediate full plane, the initial cyan intermediate full plane, and the green full plane. Specifically, the red and green intermediate full plane is summed with the green full plane to obtain a red full plane; the initial cyan intermediate global plane is summed with the green global plane to obtain the blue global plane.
Wherein each pixel in the red global plane has a red component value and each pixel in the blue global plane has a blue component value. Thus, the default pixels in both the original red plane and the original blue plane are filled.
Further in a specific implementation of step S107, a color image is synthesized from the green full plane, the red full plane, and the blue full plane. Each pixel in the color image is provided with a red component value, a green component value, and a blue component value.
In the embodiment of the invention, because the original green channel information in the Bayer image has a better signal-to-noise ratio, the green full plane can be obtained by interpolation first. And calculating an initial red-green intermediate plane and an initial blue-green intermediate plane by utilizing the green component values of all the pixels in the green full plane and the original red plane and the original blue plane. The intermediate full plane can be obtained by performing interpolation and completion on the two intermediate planes, and then the red full plane and the blue full plane are obtained by calculation, so that the color image of each pixel with red, green and blue component values can be obtained. Because the green full plane is used as the reference information for calculating and complementing the middle plane, the final color image has better signal-to-noise ratio, and the image quality is improved.
In one embodiment of the present invention, step S106 shown in fig. 1 may include the following steps: calculating the sum of the initial red-green middle full plane and the green full plane to obtain the red full plane; calculating a sum of the initial cyan intermediate global plane and the green global plane to obtain the blue global plane.
In this embodiment, the pixel value of the pixel in the middle full plane is R-G or B-G, and the sum of the middle full plane and the green full plane may be directly used as the red full plane or the blue full plane, so as to consider the information (such as the signal-to-noise ratio) in the green full plane to the red full plane and the blue full plane.
In another embodiment of the present invention, step S106 shown in fig. 1 may include the following steps: calculating the product of the initial red-green middle full plane and the green full plane to obtain the red full plane; calculating the product of the initial cyan intermediate full plane and the green full plane to obtain the blue full plane.
In this embodiment, the pixel value of the pixel in the middle full plane is R/G or B/G, the initial red-green middle full plane, the initial blue-green middle full plane and the green full plane may be multiplied, and the product is used as the red full plane and the blue full plane.
In another embodiment of the present invention, the pixel value of the pixel in the middle full plane is R + G or B + G, and the difference between the initial red and green middle full plane, the initial blue and green middle full plane and the green full plane can be calculated respectively, and the difference can be used as the red full plane and the blue full plane.
In an embodiment of the present invention, referring to fig. 8, step S106 shown in fig. 1 may include the following steps:
step S801: calculating an initial red full plane and an initial blue full plane according to the initial red-green middle full plane, the initial blue-green middle full plane and the green full plane;
step S802: calculating a luminance plane using the green full plane, the initial red full plane, and the initial blue full plane;
step S803: guiding and filtering the initial red-green intermediate full plane and the initial blue-green intermediate full plane by utilizing the brightness information of the brightness plane to obtain a final red-green intermediate full plane and a final blue-green intermediate full plane;
step S804: and calculating the red full plane and the blue full plane according to the final red-green intermediate full plane, the final blue-green intermediate full plane and the green full plane.
Unlike the previous embodiment, after the initial red full plane or the initial blue full plane is obtained according to the sum of the intermediate full plane and the green full plane, the information (such as the signal-to-noise ratio and detail information) in the luminance plane, specifically, the details such as flatness, edge, texture, etc., are also considered to be the red full plane and the blue full plane.
In a specific embodiment, in calculating the luminance plane, the luminance plane may be determined according to a maximum value of pixel values in the green full plane, the initial red full plane, and the initial blue full plane. That is, for the pixel (i, j) in the luminance plane, the maximum value of the green component value G of the pixel (i, j) in the green full plane, the red component value R in the initial red full plane, and the blue component value B in the initial blue full plane is selected as the pixel value of the pixel (i, j).
Alternatively, the luminance plane may be determined by calculating the average of the pixel values of the pixels at the same position in the green full plane, the initial red full plane, and the initial blue full plane. That is, for the pixel (i, j) in the luminance plane, the average of the green component value G of the pixel (i, j) in the green full plane, the red component value R in the initial red full plane, and the blue component value B in the initial blue full plane is selected as the pixel value of the pixel (i, j).
Alternatively, the luminance plane may be determined by calculating the minimum value of the pixel values of the pixels at the same positions in the green full plane, the initial red full plane, and the initial blue full plane. That is, for the pixel (i, j) in the luminance plane, the minimum value of the green component value G of the pixel (i, j) in the green full plane, the red component value R in the initial red full plane, and the blue component value B in the initial blue full plane is selected as the pixel value of the pixel (i, j).
In this embodiment, since the luminance information, whether the signal-to-noise ratio or the content information, is much better than that of a single red or blue channel, the final red-green intermediate full-plane and final blue-green intermediate full-plane information is richer, the signal-to-noise ratio is also better, and the image quality of the final color image is further improved.
When the initial red-green intermediate full plane and the initial blue-green intermediate full plane are subjected to the pilot filtering using the luminance information of the luminance plane, the luminance plane is used as the pilot map, and the initial red-green intermediate full plane or the initial blue-green intermediate full plane is used as the input map. For a more detailed embodiment of guiding the filtering, reference may be made to the description of the filtering algorithm known in the art, and the description is not repeated here.
In one non-limiting embodiment of the present invention, step S105 shown in fig. 1 may include the following steps:
and respectively guiding and filtering the initial red-green middle plane and the initial blue-green middle plane according to the green full plane.
As described above, the default pixels exist in the initial red and green middle plane and the initial blue and green middle plane, and the embodiment adopts a method of guiding filtering to the two middle planes to complete the filling of the default pixels in the middle planes, so as to obtain the middle full plane, that is, the initial red and green middle full plane and the initial blue and green middle full plane, wherein each pixel in the initial red and green middle full plane and the initial blue and green middle full plane has a color difference value.
In another non-limiting embodiment of the present invention, since default pixels exist in the initial red-green intermediate plane and the initial blue-green intermediate plane, in order to obtain the initial red-green intermediate full plane and the initial blue-green intermediate full plane, the non-default pixels in the neighborhood where the default pixels in the initial red-green intermediate plane are located may be used to calculate the pixel values of the default pixels, for example, an average value of the pixel values of the non-default pixels in the neighborhood where the default pixels are located is calculated to be used as the pixel values of the default pixels.
Similarly, the pixel value of the default pixel may also be calculated using non-default pixels in the neighborhood of the default pixel in the initial cyan mid-plane.
It should be noted that, the pixel value of the default pixel may also be calculated by using any other practicable mathematical calculation method for the non-default pixels in the neighborhood, which is not limited in the embodiment of the present invention.
In one non-limiting embodiment of the present invention, step S103 shown in fig. 1 may include the following steps: calculating the gradients of the original green plane in the horizontal direction and the vertical direction; and selecting the direction with the minimum gradient value to interpolate the default pixels of the original green plane so as to obtain the green full plane.
In this embodiment, the magnitude of the gradient in the original green plane may indicate how dense the texture is in the plane, for example, the plane may be divided into flat, textured, strong edge, weak edge, and other regions according to the magnitude of the gradient. The default pixels of the original green plane are interpolated by selecting the direction with the minimum gradient value, so that the accuracy of each pixel in the green full plane can be ensured.
In one embodiment, when interpolating the default pixel, if the default pixel is in a flat area, the pixel values of the non-default pixels in the neighborhood of the default pixel may be selected and averaged to obtain the pixel value of the default pixel. For example, for pixel (i, j), 4 green component values of the pixel (i, j) in the neighborhood of the original green plane may be selected and the average of the 4 green component values may be calculated to obtain the green component value of pixel (i, j).
In a specific embodiment, when interpolating the default pixel, if the default pixel is in the texture region, a neighborhood region of the default pixel in four directions (specifically, refer to fig. 9) may be selected, where the four directions include up, down, left, and right; determining a corresponding weight value according to the gradient value of the direction of the neighborhood region, wherein the larger the gradient value is, the smaller the weight value is; calculating the average value of the pixel values in the neighborhood region, multiplying the average value of the pixel values of each neighborhood region by the weight corresponding to the neighborhood region, and calculating the sum of the products to obtain the pixel value (namely, the green component value) of the default pixel.
Specifically, in the neighborhood region shown in fig. 9, the shaded portion shows the default pixel; the size of the neighborhood region can be adaptively adjusted according to the actual application scene, which is not arranged in the embodiment of the invention.
It should be noted that interpolation algorithms such as a nearest neighbor interpolation method, a bilinear interpolation, a bicubic interpolation, and the like may also be used to perform interpolation to obtain a green full plane, which is not limited in the embodiment of the present invention.
Referring to fig. 10, an embodiment of the present invention further discloses an image interpolation apparatus 90, where the image interpolation apparatus 90 may include:
a bayer image obtaining module 901, configured to obtain a bayer image to be interpolated;
a color channel separation module 902, configured to move a sliding window in the bayer image to be interpolated, and separate pixels in the sliding window according to a color channel to obtain each original color plane;
a green whole plane interpolation module 903, configured to interpolate on an original green plane after the sliding window is moved each time to obtain a green whole plane, where each pixel in the green whole plane has a green component value;
a middle plane calculating module 904, configured to calculate an initial red-green middle plane and an initial blue-green middle plane according to the green full plane, the original red plane, and the original blue plane after moving the sliding window each time;
a middle full plane calculation module 905, configured to interpolate the initial red-green middle plane and the initial blue-green middle plane according to the green full plane after the sliding window is moved each time, so as to obtain an initial red-green middle full plane and an initial blue-green middle full plane, where each pixel in the initial red-green middle full plane and the initial blue-green middle full plane has a color component difference value;
a red-blue full plane calculation module 906, configured to calculate a red full plane and a blue full plane according to at least the initial red-green intermediate full plane, the initial blue-green intermediate full plane, and the green full plane after moving the sliding window each time, where each pixel in the red full plane has a red component value and each pixel in the blue full plane has a blue component value;
and the color image calculation module 907 is configured to synthesize a color image according to the green full plane, the red full plane and the blue full plane after the sliding window is moved each time until the bayer image to be interpolated is traversed.
In the embodiment of the invention, because the original green channel information in the Bayer image has a better signal-to-noise ratio, the green full plane can be obtained by interpolation first. And calculating an initial red-green intermediate plane and an initial blue-green intermediate plane by utilizing the green component values of all the pixels in the green full plane and the original red plane and the original blue plane. The intermediate full plane can be obtained by performing interpolation and completion on the two intermediate planes, and then the red full plane and the blue full plane are obtained by calculation, so that the color image of each pixel with red, green and blue component values can be obtained. Because the green full plane is used as the reference information for calculating and complementing the middle plane, the final color image has better signal-to-noise ratio, and the image quality is improved.
For more details of the operation principle and the operation mode of the image interpolation apparatus 90, reference may be made to the description in fig. 1 to 9, and details are not repeated here.
The embodiment of the invention also discloses a storage medium, which stores computer instructions, and the computer instructions can execute the steps of the method shown in the figure 1 or the figure 8 when running. The storage medium may include ROM, RAM, magnetic or optical disks, etc. The storage medium may further include a non-volatile memory (non-volatile) or a non-transitory memory (non-transient), and the like.
The embodiment of the invention also discloses an image signal processor, and the terminal can comprise a memory and a processor, wherein the memory is stored with computer instructions capable of running on the processor. The processor, when executing the computer instructions, may perform the steps of the method shown in fig. 1 or fig. 8.
The embodiment of the invention also discloses a terminal which can comprise a memory and a processor, wherein the memory is stored with computer instructions capable of running on the processor. The processor, when executing the computer instructions, may perform the steps of the method shown in fig. 1 or fig. 8. The terminal includes, but is not limited to, a mobile phone, a computer, a tablet computer and other terminal devices.
Although the present invention is disclosed above, the present invention is not limited thereto. Various changes and modifications may be effected therein by one skilled in the art without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (10)
1. An image interpolation method, comprising:
obtaining a Bayer image to be interpolated;
moving a sliding window in the Bayer image to be interpolated, and separating pixels in the sliding window according to color channels to obtain each original color plane;
after moving the sliding window each time, performing interpolation on an original green plane to obtain a green full plane, wherein each pixel in the green full plane has a green component value;
after the sliding window is moved each time, calculating an initial red-green intermediate plane and an initial blue-green intermediate plane according to the green full plane, the initial red-green intermediate plane and the initial blue-green intermediate plane, wherein the initial red-green intermediate plane represents the difference between a red component and a green component in the Bayer image to be interpolated, and the initial blue-green intermediate plane represents the difference between a blue component and a green component in the Bayer image to be interpolated;
after the sliding window is moved each time, interpolating the initial red-green middle plane and the initial blue-green middle plane according to the green full plane to obtain an initial red-green middle full plane and an initial blue-green middle full plane, wherein each pixel in the initial red-green middle full plane and the initial blue-green middle full plane has a color component difference value;
after each movement of the sliding window, calculating a red full plane and a blue full plane according to at least the initial red-green intermediate full plane, the initial blue-green intermediate full plane and the green full plane, each pixel in the red full plane having a red component value and each pixel in the blue full plane having a blue component value;
and after the sliding window is moved each time, synthesizing a color image according to the green full plane, the red full plane and the blue full plane until the Bayer image to be interpolated is traversed.
2. The method of image interpolation according to claim 1, wherein said calculating red and blue holoplanes from at least the initial red-green intermediate holoplane, the initial cyan intermediate holoplane, and the green holoplane comprises:
calculating an initial red full plane and an initial blue full plane according to the initial red-green middle full plane, the initial blue-green middle full plane and the green full plane;
calculating a luminance plane using the green full plane, the initial red full plane, and the initial blue full plane;
guiding and filtering the initial red-green intermediate full plane and the initial blue-green intermediate full plane by utilizing the brightness information of the brightness plane to obtain a final red-green intermediate full plane and a final blue-green intermediate full plane;
and calculating the red full plane and the blue full plane according to the final red-green intermediate full plane, the final blue-green intermediate full plane and the green full plane.
3. The method of image interpolation according to claim 1, wherein said calculating red and blue holoplanes from at least the initial red-green intermediate holoplane, the initial cyan intermediate holoplane, and the green holoplane comprises:
calculating the sum of the initial red-green middle full plane and the green full plane to obtain the red full plane;
calculating a sum of the initial cyan intermediate global plane and the green global plane to obtain the blue global plane.
4. The image interpolation method of claim 1, wherein the interpolating the initial red-green intermediate plane and the initial blue-green intermediate plane according to the green holoplane comprises:
and respectively guiding and filtering the initial red-green middle plane and the initial blue-green middle plane according to the green full plane.
5. The image interpolation method according to claim 1, wherein the interpolating on the original green plane comprises:
calculating the gradients of the original green plane in the horizontal direction and the vertical direction;
and selecting the direction with the minimum gradient value to interpolate the default pixels of the original green plane so as to obtain the green full plane.
6. The method of image interpolation according to any one of claims 1 to 5, wherein each pixel in the color image has a red color component value, a blue color component value and a green color component value.
7. An image interpolation apparatus, characterized by comprising:
the Bayer image acquisition module is used for acquiring a Bayer image to be interpolated;
the color channel separation module is used for moving a sliding window in the Bayer image to be interpolated and separating pixels in the sliding window according to color channels to obtain each original color plane;
the green full-plane interpolation module is used for interpolating on the original green plane after the sliding window is moved every time to obtain a green full plane, and each pixel in the green full plane has a green component value;
the middle plane calculation module is used for calculating an initial red-green middle plane and an initial blue-green middle plane according to the green full plane, the original red plane and the original blue plane after the sliding window is moved each time;
the intermediate full plane calculation module is used for interpolating the initial red and green intermediate plane and the initial blue and green intermediate plane according to the green full plane after the sliding window is moved every time so as to obtain an initial red and green intermediate full plane and an initial blue and green intermediate full plane, and each pixel in the initial red and green intermediate full plane and the initial blue and green intermediate full plane has a color component difference value;
a red-blue full plane calculation module for calculating a red full plane and a blue full plane at least according to the initial red-green intermediate full plane, the initial blue-green intermediate full plane and the green full plane after moving the sliding window each time, wherein each pixel in the red full plane has a red component value, and each pixel in the blue full plane has a blue component value;
and the color image calculation module is used for synthesizing a color image according to the green full plane, the red full plane and the blue full plane after the sliding window is moved each time until the Bayer image to be interpolated is traversed.
8. A storage medium having stored thereon computer instructions which, when executed, perform the steps of the image interpolation method of any of claims 1 to 6.
9. An image signal processor comprising a memory and a processor, the memory having stored thereon computer instructions executable on the processor, wherein the processor, when executing the computer instructions, performs the steps of the image interpolation method of any one of claims 1 to 6.
10. A terminal comprising a memory and a processor, the memory having stored thereon computer instructions executable on the processor, wherein the processor, when executing the computer instructions, performs the steps of the image interpolation method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911121851.0A CN110852953B (en) | 2019-11-15 | 2019-11-15 | Image interpolation method and device, storage medium, image signal processor and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911121851.0A CN110852953B (en) | 2019-11-15 | 2019-11-15 | Image interpolation method and device, storage medium, image signal processor and terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110852953A true CN110852953A (en) | 2020-02-28 |
| CN110852953B CN110852953B (en) | 2022-09-02 |
Family
ID=69600949
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911121851.0A Active CN110852953B (en) | 2019-11-15 | 2019-11-15 | Image interpolation method and device, storage medium, image signal processor and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110852953B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113454687A (en) * | 2020-09-28 | 2021-09-28 | 深圳市大疆创新科技有限公司 | Image processing method, apparatus and system, computer readable storage medium |
| CN113824936A (en) * | 2021-09-23 | 2021-12-21 | 合肥埃科光电科技有限公司 | Color interpolation method, device and equipment for color filter array line scanning camera |
| CN115689887A (en) * | 2022-10-28 | 2023-02-03 | 辉羲智能科技(上海)有限公司 | Automatic driving RGBIR image resampling method, system, terminal and medium |
| CN116245753A (en) * | 2022-12-30 | 2023-06-09 | 北京华云星地通科技有限公司 | Red and blue stereoscopic satellite cloud image generation method, system, electronic equipment and medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101917629A (en) * | 2010-08-10 | 2010-12-15 | 浙江大学 | Green component and color difference space-based Bayer format color interpolation method |
| CN104537625A (en) * | 2015-01-05 | 2015-04-22 | 中国科学院光电技术研究所 | Bayer color image interpolation method based on direction flag bits |
| CN105577981A (en) * | 2015-12-22 | 2016-05-11 | 深圳大学 | Edge self-adaptive color restoration method and system |
| CN108734668A (en) * | 2017-04-21 | 2018-11-02 | 展讯通信（上海）有限公司 | Image color restoration methods, device, computer readable storage medium and terminal |
| US20190318451A1 (en) * | 2018-04-11 | 2019-10-17 | Omnivision Technologies, Inc. | Image demosaicer and method |
-
2019
- 2019-11-15 CN CN201911121851.0A patent/CN110852953B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101917629A (en) * | 2010-08-10 | 2010-12-15 | 浙江大学 | Green component and color difference space-based Bayer format color interpolation method |
| CN104537625A (en) * | 2015-01-05 | 2015-04-22 | 中国科学院光电技术研究所 | Bayer color image interpolation method based on direction flag bits |
| CN105577981A (en) * | 2015-12-22 | 2016-05-11 | 深圳大学 | Edge self-adaptive color restoration method and system |
| CN108734668A (en) * | 2017-04-21 | 2018-11-02 | 展讯通信（上海）有限公司 | Image color restoration methods, device, computer readable storage medium and terminal |
| US20190318451A1 (en) * | 2018-04-11 | 2019-10-17 | Omnivision Technologies, Inc. | Image demosaicer and method |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113454687A (en) * | 2020-09-28 | 2021-09-28 | 深圳市大疆创新科技有限公司 | Image processing method, apparatus and system, computer readable storage medium |
| WO2022061879A1 (en) * | 2020-09-28 | 2022-03-31 | 深圳市大疆创新科技有限公司 | Image processing method, apparatus and system, and computer-readable storage medium |
| CN113824936A (en) * | 2021-09-23 | 2021-12-21 | 合肥埃科光电科技有限公司 | Color interpolation method, device and equipment for color filter array line scanning camera |
| CN113824936B (en) * | 2021-09-23 | 2024-02-09 | 合肥埃科光电科技股份有限公司 | Color interpolation method, device and equipment for color filter array line scanning camera |
| CN115689887A (en) * | 2022-10-28 | 2023-02-03 | 辉羲智能科技(上海)有限公司 | Automatic driving RGBIR image resampling method, system, terminal and medium |
| CN116245753A (en) * | 2022-12-30 | 2023-06-09 | 北京华云星地通科技有限公司 | Red and blue stereoscopic satellite cloud image generation method, system, electronic equipment and medium |
| CN116245753B (en) * | 2022-12-30 | 2023-10-03 | 北京华云星地通科技有限公司 | Red and blue stereoscopic satellite cloud image generation method, system, electronic equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110852953B (en) | 2022-09-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110852953B (en) | Image interpolation method and device, storage medium, image signal processor and terminal | |
| US10467735B2 (en) | Inverse tone mapping based on luminance zones | |
| US9654700B2 (en) | Computational camera using fusion of image sensors | |
| US9344639B2 (en) | High dynamic range array camera | |
| JP5672776B2 (en) | Image processing apparatus, image processing method, and program | |
| US8948506B2 (en) | Image processing device, image processing method, and program | |
| KR20010032809A (en) | A New Edge-Detection Based Noise Removal Algorithm | |
| KR101637488B1 (en) | Image interpolation method and apparatus using pattern characteristics of color filter array | |
| CN108734668B (en) | Image color recovery method and device, computer readable storage medium and terminal | |
| CN108122201A (en) | A kind of Bayer interpolation slide fastener effect minimizing technology | |
| CN111539893A (en) | Bayer image joint demosaicing denoising method based on guided filtering | |
| TWI544785B (en) | Image downsampling apparatus and method | |
| US8346019B2 (en) | Image generation method and apparatus, program therefor, and storage medium which stores the program | |
| CN105430357A (en) | Demosaicing method and device for image sensor | |
| CN113454687A (en) | Image processing method, apparatus and system, computer readable storage medium | |
| Saito et al. | Demosaicing approach based on extended color total-variation regularization | |
| CN114359050B (en) | Image processing method, apparatus, computer device, storage medium, and program product | |
| KR101327790B1 (en) | Image interpolation method and apparatus | |
| JP3899144B2 (en) | Image processing device | |
| US8565313B2 (en) | Determining a vector field for an intermediate image | |
| KR20150094350A (en) | Method and apparatus for executing demosaicking using edge information | |
| JPH0490287A (en) | Filter device | |
| JP4139587B2 (en) | Interpolation apparatus and method for captured image in single-plate color digital camera | |
| US8442359B2 (en) | Image interpolation method and apparatus using reference block based on direction | |
| US20050190993A1 (en) | Method and apparatus for determining an edge trend for an interested pixel of an image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |