CN108156461B

CN108156461B - Bayer image compression method and device

Info

Publication number: CN108156461B
Application number: CN201711455811.0A
Authority: CN
Inventors: 林江; 陈本强; 陈涛; 王洪剑
Original assignee: Shanghai Tongtu Semiconductor Technology Co ltd
Current assignee: Qingchi (Jinan) Intelligent Technology Co.,Ltd.
Priority date: 2017-12-28
Filing date: 2017-12-28
Publication date: 2021-12-10
Anticipated expiration: 2037-12-28
Also published as: CN108156461A

Abstract

The invention discloses a Bayer image compression method and a Bayer image compression device, wherein the method comprises the following steps: step S1, performing macroblock segmentation on the input single-line image; step S2, predicting the current macro block to obtain the prediction residual error of the current pixel point; step S3, carrying out noise analysis through the prediction residual error and the gradient information to obtain a noise level; step S4, carrying out self-adaptive filtering on the reconstructed pixel points in the line buffer area by using the obtained noise level; and step S5, entropy coding the prediction residual error and outputting a code stream, wherein the invention analyzes the noise level of the image and carries out self-adaptive filtering processing aiming at the images with different noise levels, thereby achieving the purposes of effectively inhibiting noise, greatly improving the compression efficiency and effectively retaining the real original information of the image.

Description

Bayer image compression method and device

Technical Field

The invention relates to the technical field of image processing, in particular to a Bayer image compression method and device based on noise and other source characteristic analysis.

Background

Currently, most cameras and digital cameras use CCD or CMOS sensors for light sensing, and in order to obtain Color information, a two-dimensional Color Filter Array CFA (Color Filter Array) is mostly placed between the lens and the sensor. Each filter element of the CFA corresponds to a sensor pixel, and light is projected on the sensor through the CFA. Each pixel cell of the sensor collects R, G or B intensity information for only one color. Thus, the Bayer array (fig. 1) becomes the most widely used CFA pixel distribution format at present, i.e., the Bayer format image becomes the final output format of the sensor.

However, due to process and technical reasons, the above sensor imaging has more or less noise problems, the performance is not perfect, and especially under dark light conditions, the noise problem of the image sensor is more prominent, so that the image directly obtained from the sensor has considerable difference from the actual image in details and colors.

At this time, ISP (Image Signal Processing) is required to perform post-Processing on the Image from the sensor, and these Processing algorithms include linear correction, noise removal, dead pixel removal, interpolation, automatic white balance, automatic exposure, etc. in order to complete all Image Processing, Image data of one frame or several continuous frames is often required, so that the Bayer Image needs to be stored in DDR after the sensor is imaged, so as to be convenient for reading.

Currently, compression methods for Bayer images are mainly classified into the following three categories:

(1) the bayer image is directly compressed by using the existing image compression algorithm (such as JPEG2000, JPEG-LS, and the like) or data compression algorithm (such as SPIHT, FELICS, CALIC, and the like).

(2) The RGB three components of the Bayer image are subjected to color gamut conversion, such as conversion to a YUV color gamut, certain redundant information can be eliminated, and then the RGB three components are compressed by using the existing compression algorithm.

(3) The Bayer image format is processed in the aspects of space rearrangement, matrix transformation and the like, partial redundant information can be eliminated, and then the Bayer image format is compressed by using the existing compression algorithm.

However, the above prior art mainly has the following three disadvantages:

(1) due to the fact that factors such as low illumination, unstable current and interference among circuits of various components affect the generated noisy Bayer image, even if methods such as transformation and spatial processing are adopted, noise information always occupies most information, and the effect of eliminating redundancy cannot be achieved;

(2) because the noise generally belongs to high-frequency information and is not easy to compress, if lossless compression is adopted, the expected bandwidth and storage requirements can not be met, and even side effects can be caused;

(3) if lossy compression is adopted, in order to meet the bandwidth and storage requirements, the corresponding quantization loss tends to increase sharply, that is, the original information of the image is greatly lost.

Disclosure of Invention

In order to overcome the defects of the prior art, the present invention provides a Bayer image compression method and apparatus, so as to perform noise level analysis on an image, and perform adaptive filtering processing on images with different noise levels, thereby achieving the purposes of effectively suppressing noise, greatly improving compression efficiency, and effectively retaining the real original information of the image.

To achieve the above and other objects, the present invention provides a Bayer image compression method, including the steps of:

step S1, performing macroblock segmentation on the input single-line image;

step S2, predicting the current macro block to obtain the prediction residual error of the current pixel point;

step S3, carrying out noise analysis through the prediction residual error and the gradient information to obtain a noise level;

step S4, carrying out self-adaptive filtering on the reconstructed pixel points in the line buffer area by using the obtained noise level;

and step S5, entropy coding the prediction residual error and outputting a code stream.

Preferably, in step S2, the same prediction algorithm is used for the R/G/B three components, and the relationship between the current pixel point and the surrounding pixel points is used to obtain the predicted pixel point value, so as to obtain the predicted residual error of the current pixel point.

Preferably, the step S2 further includes:

reading out a reconstruction point pix _ b, a reconstruction point pix _ c and a reconstruction point pix _ d of the previous line from the line cache region, wherein pix _ b is a reconstruction pixel point right above, pix _ c is a reconstruction pixel point at the upper left, and pix _ d is a reconstruction pixel point at the upper right, and reading out a left original pixel point pix _ a and a left original pixel point pix _ f at the left end;

filtering the left-end pixel pix _ a and the left-end pixel pix _ f to obtain a filtered pixel pix _ flt;

calculating and solving the maximum value max _ ab and the minimum value min _ ab of the left pixel point and the pixel point right above;

obtaining a final predicted pixel point value according to the relation between the upper left reconstructed pixel point pix _ c and the calculation result;

and calculating the prediction residual error according to the prediction pixel point and the current pixel point.

Preferably, the step S3 further includes:

analyzing the current pixel point and surrounding pixel points to obtain gradient information of the current pixel point;

carrying out histogram statistics on the gradient of the current pixel point, and carrying out classification processing;

finding the class with the minimum gradient in all current classes, wherein the number of the pixel points is grad _ bin [0], and further obtaining the proportion sgrad _ ratio of the gradient in all the pixel points of the whole image;

and configuring different threshold values according to the ratio sgrad _ ratio, and further calculating the noise level of the current image.

Preferably, the noise level is obtained by the following formula:

wherein thr1, thr2, thr3 and thr4 are threshold parameters for determining noise level.

Preferably, the step of analyzing the current pixel point and the surrounding pixel points to obtain the gradient information of the current pixel point specifically includes: calculating to obtain gradients between the current pixel pix _0 and surrounding pixels pix _ a, pix _ b, pix _ c and pix _ d, and taking the minimum value between the four gradients as the final gradient of the current pixel.

Preferably, in step S4, for the adaptive filtering of the G component, step S4 specifically includes:

selecting a filter of even order;

selecting a current reconstruction pixel point, two reconstruction pixel points on the left side of the current reconstruction pixel point and three reconstruction pixel points on the right side of the current reconstruction pixel point;

carrying out self-adaptive filtering by using the selected filter;

normalizing the filtered reconstruction pixel points to obtain final reconstruction pixel points;

and updating the reconstructed pixel point of the current macro block into a line buffer area.

Preferably, in step S4, for the adaptive filtering of the R/B component, step S4 specifically includes:

selecting an odd-order filter;

selecting a current reconstruction pixel point, three reconstruction pixel points on the left side and three reconstruction pixel points on the right side;

carrying out self-adaptive filtering;

Preferably, the step S5 further includes:

post-processing the predicted residual to obtain a residual coefficient to be finally coded;

and entropy coding the residual error coefficient and outputting a code stream.

To achieve the above object, the present invention also provides a Bayer image compression apparatus comprising:

a macroblock dividing unit that performs macroblock division on an input one-line image;

the prediction unit is used for predicting the current macro block to obtain the prediction residual error of the current pixel point;

the noise analysis unit is used for carrying out noise analysis according to the prediction residual error and the gradient information to obtain a noise level;

the adaptive filtering unit is used for carrying out adaptive filtering on the reconstructed pixel points in the line cache region by utilizing the obtained noise level;

and the entropy coding unit is used for entropy coding the prediction residual error and outputting a code stream.

Compared with the prior art, the Bayer image compression method and the Bayer image compression device perform noise level analysis on the image and perform adaptive filtering processing on the image with different noise levels, so that the purposes of effectively suppressing noise, greatly improving compression efficiency and effectively retaining real original information of the image can be achieved.

Drawings

FIG. 1 is a schematic diagram of a current Bayer format image;

FIG. 2 is a flow chart of the steps of a Bayer image compression method of the present invention;

FIG. 3 is a schematic diagram illustrating a positional relationship between a current pixel point and surrounding pixel points according to an embodiment of the present invention;

FIG. 4 is a detailed flowchart of step S2 according to an embodiment of the present invention;

FIG. 5 is a detailed flowchart of step S3 according to an embodiment of the present invention;

FIG. 6 is a detailed flowchart of step S5 according to the embodiment of the present invention;

fig. 7 is a system architecture diagram of a Bayer image compression apparatus.

Detailed Description

Other advantages and capabilities of the present invention will be readily apparent to those skilled in the art from the present disclosure by describing the embodiments of the present invention with specific embodiments thereof in conjunction with the accompanying drawings. The invention is capable of other and different embodiments and its several details are capable of modification in various other respects, all without departing from the spirit and scope of the present invention.

FIG. 2 is a flow chart of the steps of a Bayer image compression method of the present invention. As shown in fig. 2, a Bayer image compression method of the present invention includes the following steps:

in step S1, Macroblock (MB) segmentation is performed on the input single-line image, and specifically, the segmented macroblock size may be 1, 16, 32, 64, 128, or the like. Since the specific macroblock partition is the prior art, it is not described herein.

Step S2, predict the current macroblock to obtain the prediction residual pred _ diff of the current pixel.

In the embodiment of the invention, the same prediction algorithm is used for the R/G/B three components, and the relation between the current pixel pix _0 and the surrounding pixels is mainly utilized to obtain the R/G/B three components. The predicted point of the upper line is the reconstructed point of the upper line read out from linebuffer, and the left point is the original pixel point. The specific positional relationship is shown in fig. 3. Specifically, as shown in fig. 4, step S2 further includes:

step S200, reading out the reconstruction points pix _ b, pix _ c and pix _ d of the previous line from the line buffer linebuffer, wherein pix _ b is a reconstruction pixel point right above, pix _ c is a reconstruction pixel point at the upper left, and pix _ d is a reconstruction pixel point at the upper right; reading out a left original pixel pix _ a and a left original pixel pix _ f;

step S201, filtering the left-end pixel pix _ a and the left-end pixel pix _ f to obtain a filtered pixel pix _ flt; the formula is as follows:

where flt _ coeff (,) represents a filter coefficient, and different noise levels noise _ lev correspond to different filter coefficients.

Step S202, calculating and solving the maximum and minimum values of the left pixel point and the right pixel point; the formula is as follows:

max_ab＝MAX(pix_flt,pix_b)

min_ab＝MIN(pix_flt,pix_b)

where, MAX (,) represents taking the maximum of the two, MIN (,) represents taking the minimum of the two.

Step S203, solving a final predicted pixel point value according to the relation between the upper left reconstructed pixel point pix _ c and the calculation result in the step S202;

in step S204, a prediction residual is calculated. Specifically, the difference between the predicted pixel pred and the current pixel pix _0 is made, so as to obtain a predicted residual pred _ diff. The formula is as follows:

pred_diff＝pix_0-pred

and step S3, carrying out noise analysis through information such as prediction residual error and gradient, and obtaining the noise level.

Specifically, in step S3, by analyzing information such as the gradient of the current pixel pix _0 and surrounding pixels, corresponding statistical information can be obtained, so as to obtain a noise level of one frame, which is divided into five noise levels in total, where noise level noise _ lev is 4 as the strongest noise level, and noise level noise _ lev is 0 as the weakest noise level. As shown in fig. 5, step S3 further includes:

step S300, calculating and obtaining gradients between the current pixel pix _0 and surrounding pixel points pix _ a, pix _ b, pix _ c and pix _ d, and taking the minimum value between the four gradients as the final gradient of the current pixel; the formula is as follows:

grad＝MIN(|pix_0-pix_a|,|pix_0-pix_b|,|pix_0-pix_c|,|pix_0-pix_d|)；

wherein, | represents taking an absolute value.

Step S301, performing histogram statistics on the gradient of the current pixel point, and performing classification processing, in the specific embodiment of the present invention, the total number of categories is 256. The formula is as follows:

grad_bin[grad]++；

wherein, grad _ bin [ grad ] is the category obtained by classifying according to the grad value, if the grad is the same value, the number can be accumulated

Step S302, finding the class with the minimum gradient in all current classes, wherein the number of pixel points is grad _ bin [0], and further obtaining the proportion grad _ ratio of the pixel points in the whole image; the formula is as follows

sgrad_ratio＝grad_bin[0]/(frm_height*frm_width)；

Wherein frm _ height and frm _ width are the height and width of the image, respectively.

In step S303, different thresholds are configured according to the sgrad _ ratio, and the noise level to the current image is calculated. The specific judgment formula is as follows:

And step S4, performing self-adaptive filtering on the linebuffer according to the noise level.

That is to say, according to the noise level noise _ lev obtained in step S3, the corresponding filter may be adaptively selected, and the special filtering processing is performed on the reconstructed pixel in linebuffer to eliminate the noise redundant information and retain the effective original image information.

Because of the inconsistency and the particularity of the G component and the R/B component in space and phase, the difference processing is carried out, the redundancy in space can be effectively eliminated, the prediction accuracy is higher, and the compression efficiency can be greatly improved.

For the adaptive filtering of the G component, the specific steps are as follows:

a. and selecting a corresponding filter. For the G component, the current pixel point and the previous row of pixel points have an interlaced relation, i.e. no pixel point is corresponding to the current pixel point, so the invention selects the filter with even order, and the filter length can be selected from 2, 4, 6, 8, etc. Here, a filter with 6 th order coefficients is taken as an example.

b. And selecting pixel points. And selecting the current reconstruction pixel point, two reconstruction pixel points on the left side and three reconstruction pixel points on the right side. Respectively denoted as pix _ rec _ g (0), pix _ rec _ g (1), pix _ rec _ g (2), pix _ rec _ g (3), pix _ rec _ g (4), and pix _ rec _ g (5).

c. And carrying out self-adaptive filtering to obtain a filtered value G _ rec _ flt. The formula is as follows:

where coeff _ g (,) represents an even-order filter coefficient adaptively selected according to the noise level noise _ lev.

d. And normalizing the filtered reconstruction pixel points to obtain the final reconstruction pixel point G _ rec _ norm. The formula is as follows:

G_rec_norm＝(G_rec_flt+128)/256

e. and updating the reconstructed pixel point of the current macro block into a linebuffer. The concrete formula is as follows:

pix_rec＝pred+pred_diff

wherein pix _ rec represents the reconstructed pixel.

For R/B component adaptive filtering, the specific steps are as follows:

a. and selecting a filter. For R/B component, although the present pixel point and the last line pixel point have interlaced relation, i.e. the positions are corresponding, the invention selects the filter with odd order, the length of the filter can be selected from 1, 3, 5, 7, etc. Here, a filter with 7 th order coefficients is taken as an example.

b. And selecting pixel points. And selecting the current reconstruction pixel point, the left three reconstruction pixel points and the right three reconstruction pixel points. Respectively denoted as pix _ rec _ rb (0), pix _ rec _ rb (1), pix _ rec _ rb (2), pix _ rec _ rb (3), pix _ rec _ rb (4), pix _ rec _ rb (5), and pix _ rec _ rb (6).

c. And carrying out self-adaptive filtering to obtain a filtered value R/B _ rec _ flt. The formula is as follows:

where coeff _ rb (,) represents a filter coefficient adaptively selected according to the noise level noise _ lev.

d. And normalizing the filtered reconstruction pixel points to obtain the final reconstruction pixel point R/B _ rec _ norm. The formula is as follows:

R/B_rec_norm＝(R/B_rec_flt+128)/256

pix_rec＝pred+pred_diff

wherein pix _ rec represents the reconstructed pixel.

Step S5, entropy coding the prediction residual pred _ diff and outputting a code stream, in an embodiment of the present invention, Huffman or Golomb coding may be adopted. Specifically, as shown in fig. 6, step S5 further includes:

step S500, the prediction residual pred _ Diff is post-processed to obtain the residual coefficient Diff _ q which needs to be encoded finally.

Diff_q＝ABS(pred_diff)*2-sign(pred_diff)

Wherein sign (pred _ diff) is 1 when pred _ diff is a negative number, otherwise is 0;

in step S501, residual coefficients Diff _ q are encoded using Huffman/Golomb.

Fig. 7 is a system architecture diagram of a Bayer image compression apparatus according to the present invention. As shown in fig. 7, a Bayer image compression apparatus of the present invention includes:

the macroblock dividing unit 701 performs Macroblock (MB) division on the input single-line image, and specifically, the size of the divided macroblock may be 1, 16, 32, 64, 128, and the like, which is not limited in the present invention.

The prediction unit 702 is configured to predict a current macroblock to obtain a prediction residual pred _ diff of a current pixel.

In the embodiment of the invention, the same prediction algorithm is used for the R/G/B three components, and the relation between the current pixel pix _0 and the surrounding pixels is mainly utilized to obtain the R/G/B three components. The predicted point of the upper line is the reconstructed point of the upper line read from linebuffer (single-port line cache), and the left point is the original pixel point. The prediction unit 702 is specifically configured to:

reading out a reconstruction point pix _ b, a reconstruction point pix _ c and a reconstruction point pix _ d of the previous line from the linebuffer, wherein the pix _ b is a reconstruction pixel point right above, the pix _ c is a reconstruction pixel point at the upper left, and the pix _ d is a reconstruction pixel point at the upper right; reading out a left original pixel pix _ a and a left original pixel pix _ f;

calculating and solving the maximum and minimum values of the left pixel point and the right pixel point;

calculating a final predicted pixel point value according to the relation between the upper left reconstructed pixel point pix _ c and the calculation result;

and calculating a prediction residual error, specifically, subtracting the prediction pixel pred from the current pixel pix _0 to obtain the prediction residual error pred _ diff.

And the noise analysis unit 703 is configured to perform noise analysis according to information such as the prediction residual and the gradient, so as to obtain a noise level.

Specifically, the noise analyzing unit 703 may obtain corresponding statistical information by analyzing information such as gradients of the current pixel pix _0 and surrounding pixels, so as to obtain a noise level of one frame, which is divided into five levels in total, where noise level noise _ lev is 4, which is the strongest noise level, and noise level noise _ lev is 0, which is the weakest noise level. The noise analysis unit 403 is specifically configured to:

calculating to obtain gradients between the current pixel pix _0 and surrounding pixel points pix _ a, pix _ b, pix _ c and pix _ d, and taking the minimum value between the four gradients as the final gradient of the current pixel;

different thresholds are configured according to the sgrad _ ratio, and then the noise level of the current image is calculated.

And the adaptive filtering unit 704 is used for performing adaptive filtering on the linebuffer according to the noise level.

That is to say, according to the noise level noise _ lev obtained by the noise analysis unit 703, a corresponding filter is adaptively selected, and a special filtering process is performed on the reconstructed pixel point in the line buffer (linebuffer) to eliminate noise redundant information and retain effective original image information.

For the G component adaptive filtering, the adaptive filtering unit 404 is specifically configured to:

c. And carrying out self-adaptive filtering to obtain a filtered value G _ rec _ flt.

d. And normalizing the filtered reconstruction pixel points to obtain the final reconstruction pixel point G _ rec _ norm.

e. And updating the reconstructed pixel point of the current macro block into a linebuffer.

For R/B component adaptive filtering, the adaptive filtering unit 404 is specifically configured to:

c. And carrying out self-adaptive filtering to obtain a filtered value R/B _ rec _ flt.

d. And normalizing the filtered reconstruction pixel points to obtain the final reconstruction pixel point R/B _ rec _ norm.

The entropy coding unit 705 is configured to entropy code the prediction residual pred _ diff and output a code stream, and in an embodiment of the present invention, the entropy coding unit 705 may use Huffman (Huffman) or Golomb (Golomb) coding. The entropy coding unit 705 is specifically configured to:

and performing post-processing on the prediction residual error pred _ Diff to obtain a residual error coefficient Diff _ q which needs to be encoded finally.

The residual coefficients Diff _ q are encoded using Huffman/Golomb.

In summary, the Bayer image compression method and apparatus of the present invention perform noise level analysis on an image, and perform adaptive filtering processing on images with different noise levels, so as to achieve the purposes of effectively suppressing noise, greatly improving compression efficiency, and effectively retaining real original information of the image.

The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Modifications and variations can be made to the above-described embodiments by those skilled in the art without departing from the spirit and scope of the present invention. Therefore, the scope of the invention should be determined from the following claims.

Claims

1. A Bayer image compression method comprises the following steps:

step S1, performing macroblock segmentation on the input single-line image;

step S2, predicting the current macro block, adopting the same prediction algorithm for R/G/B three components, obtaining the predicted pixel point value by using the relation between the current pixel point and the surrounding pixel points, and further obtaining the predicted residual error of the current pixel point:

filtering the left-end pixel pix _ a and the left-end pixel pix _ f to obtain a filtered left-side pixel pix _ flt;

and obtaining a final predicted pixel point value according to the relation between the upper left reconstructed pixel point pix _ c and the calculation result:

calculating the prediction residual error according to the prediction pixel point and the current pixel point;

step S3, performing noise analysis through the prediction residual and the gradient information to obtain a noise level, including:

configuring different threshold values according to the ratio sgrad _ ratio, and further calculating the noise level of the current image;

2. The Bayer image compression method according to claim 1, wherein the step of analyzing the current pixel point and surrounding pixel points to obtain gradient information of the current pixel point specifically includes: calculating to obtain gradients between the current pixel pix _0 and surrounding pixels pix _ a, pix _ b, pix _ c and pix _ d, and taking the minimum value between the four gradients as the final gradient of the current pixel.

3. The Bayer image compression method according to claim 1, wherein, in step S4, for the G component adaptive filtering, step S4 specifically includes:

selecting a filter of even order;

carrying out self-adaptive filtering by using the selected filter;

4. The Bayer image compression method as claimed in claim 3, wherein in step S4, for the R/B component adaptive filtering, step S4 specifically includes:

selecting an odd-order filter;

carrying out self-adaptive filtering;

5. The Bayer image compression method according to claim 4, wherein the step S5 further includes:

and entropy coding the residual error coefficient and outputting a code stream.

6. A Bayer image compression apparatus comprising:

the prediction unit is used for predicting the current macro block, adopting the same prediction algorithm for the R/G/B three components, obtaining a predicted pixel point value by utilizing the relation between the current pixel point and surrounding pixel points, and further obtaining a predicted residual error of the current pixel point:

a noise analysis unit, configured to perform noise analysis according to the prediction residual and the gradient information to obtain a noise level, including: