JP2008124530A - Raw data compressing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a RAW data compressing method capable of not only compressing RAW data obtained by a single CCD solid imaging element having a CFA at a high rate and with a low distortion but also attaining a preview function to confirm the compressed RAW data conveniently as an image by a JPEG decoder. <P>SOLUTION: The RAW data observed by the single CCD solid imaging element having the CFA is directly compressed as "JPEG data capable of decoding the RAW data". The decoding process and the down sampling of the JPEG decoder are formulated. The JPEG data, which corresponds to such a quantization DCT coefficient as to reduce the error between the decoded RAW data and the observed RAW data is generated as the "JPEG data capable of decoding the RAW data". Computations to determine the quantized DCT coefficient are repeated to reduce the parameter number of the DCT coefficient. The quantization table of the DCT coefficient is redesigned to generate the "JPEG data capable of decoding the RAW data". <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a RAW data compression method for compressing RAW data acquired from a single-plate color image sensor having a color filter array (CFA).

  In recent years, digital cameras have been widely used. Among them, for many low-cost color digital cameras, a single-plate color image sensor having a color filter array (CFA) is used. The Bayer pattern is most widely used as a color filter array (CFA) (see Patent Document 1).

  In a single-plate color imaging device having a Bayer pattern color filter array, one pixel can only observe one color of R, G, or B.

  In other words, the output of such a single-plate color image pickup device having a CFA has one color value for one pixel and can be considered as one gray image, although the color channels are different.

  Raw digital image data output (observed) from such a single-plate color image pickup device having CFA is called “RAW data”, and “color filter array data (color filter array data) ", or" CFA data ".

  Here, the terms “RAW data” and “CFA data” are used interchangeably. In other words, “RAW data” or “CFA data” is output (acquired) from a single-plate color imaging device equipped with CFA. ) Means raw digital image data.

  Image processing for generating a full-color image from RAW data, that is, converting RAW data into a full-color image is called “demosaicing processing”, and many methods have been proposed for demosaicing processing in the past ( (Refer nonpatent literature 1).

  Here, a standard procedure from capturing an image using a digital camera using a single-plate color image sensor equipped with CFA to displaying the captured image will be described.

  In general, “RAW data” observed (output) by a single-plate color imaging device equipped with a CFA is first converted to “full color image data” (made full color) by demosaicing processing. Next, the converted “full color image data” is, for example, JPEG (see Non-Patent Document 2), JPEG-LS (see Non-Patent Document 3), or JPEG-2000 (see Non-Patent Document 4). Data compression is performed by the image compression method and is stored as “image data”.

  Finally, the stored “image data” is restored to a full-color image by a decoder corresponding to the compression encoder that has performed data compression, and displayed on a display such as a liquid crystal display.

  As described above, in a digital camera using a single-plate color imaging device equipped with a CFA, RAW data is compressed until a full color image is displayed from RAW data. However, this is not a method for directly compressing RAW data. Absent.

  In recent years, in image processing for generating a full-color image from RAW data, first, RAW data acquired (output) from a single-plate color image sensor equipped with CFA is stored, and when necessary, A technique called “RAW development” is also being used in which a full color image is generated by performing image processing such as demosaicing processing on stored RAW data with an information processing apparatus such as a personal computer.

Furthermore, even in “color super-resolution processing”, which is image processing for generating one color high-resolution image from a plurality of low-resolution images, super-resolution processing is effectively performed by using RAW data for reconstruction. (See Non-Patent Document 5).
U.S. Patent No. 3971065 BK Gunturk, J. Glotzbach, Y. Altunbasak, RW Schafer, RM Mersereau, " Demosaiking: Color filter array interpolation ”, IEEE Signal Processing Magazine, Vol. 22, No. 1, p. 44-54, 2005 G. K. Wallace, “The JPEG still picture compression standard”, Communications of the ACM, Vol. 34, No. 4, p. 30-40, 1991 M. J. Weinberger, G. Seroussi, G. Sapiro, "The LOCO-I Lossless Image Compression Algorithm: Principal and Standardization Into JPEG-LS (The LOCO-I lossless image compression algorithm: principles and standardization into JPEG-LS), IEEE Trans. Image Processing, Vol. 9, No. 8, p. 1309-1324, 2000 A. Skodras, C. Christopoulos, T. Ebrahimi, "The JPEG 2000 still image compression standard", IEEE signal Processing Magazine (IEEE Signal Processing Magazine), Vol.18, No.5, p.36-58, 2001 T. Goto and M. Okutomi, “Direct Super-Resolution and Registration Using Raw CFA Images”, Plock. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition), Volume II, p.600-607, 2004 Co-authored by N. Zhang and X. Wu, “Lossless Compression of Color Mosaice Images”, IEEE Trans. Image Processing, IEEE Trans. Image Processing Vol.15, No.6, p.1379-1388, 2006 S.-Y. Lee and A. Ortega, “A Novel Approach of Image Compression in Digital Cameras with a Bayer Color Filter Array ”, Proc. International Conference of Image Processing, Volume 3, p.482-485, 2001 Co-authored by CC Koh, J. Mukherjee, and SK Mitra, “New Efficient Methods Off Image Compression in Digital Camera with Color Filter Array” Compression in Digital Cameras with Color Filter Array), Trans. Consumer Electronics, Vol. 49, No. 4, pp.1488-1456, 2003 X. Xie, G. L. Li (GL Li), Zed H. Wang, C. Zhang, D. M. Li (DM Li), X. Co-authored by XW Li, "A Novel Method of Lossy Image Compression for Digital Image Sensors with Bayer Color Filter Arrays", Plock. International Symposium Proc. International Symposium on Circuits and Systems, Vol. 5, p. 4995-4998, 2005 JE Adams, JF Hamilton, "Design of practical color filter array interpolation algorithms for digital cameras" Proc. SPIE, 3028, p.117-125, 1997 K. Hirakawa, T. W. Parks, “Adaptive homogeneity-directed demosaicing algorithm”, IEEE Trans. Image Processing (IEEE Trans. Image Processing) Processing, Vol. 14, No. 3, p. 360-369, 2005

  However, a method for efficiently storing RAW data obtained from a single-plate color image sensor equipped with CFA and an appropriate storage format for RAW data have not been established. In many cases, RAW data is uncompressed. It is currently preserved as it is.

  When saving raw data without compression, there is a problem that the storage format is not unified. Another problem is that the data capacity is very large due to the non-compression method.

  Furthermore, in order to confirm the RAW data stored uncompressed as an image, a decoder and a demosaicing process corresponding to each uncompressed storage method are required. Therefore, there is a problem that it is not possible to simply check RAW data stored uncompressed as an image using an existing decoder.

  On the other hand, in order to compress and store a full-color image, for example, a compression storage format such as JPEG, JPEG-LS, and JPEG-2000 can be used. In particular, JPEG is very popular and is said to be one of the de facto standards for image compression formats.

  In view of this, several methods have been proposed as RAW data compression methods in recent years using compression storage formats such as JPEG, JPEG-LS, and JPEG-2000 (Non-Patent Documents 6, 7, 8, and 9). See).

  In any of these RAW data compression methods, RAW data is separated for each color channel, and then the image of the separated color channel is converted into a gray image, and image compression such as existing JPEG, JPEG-LS, or JPEG-2000 is performed. There is a common point in applying the method. By using these RAW data compression methods, it is possible to compress and store RAW data with high compression and low distortion.

  However, in these RAW data compression methods, in order to confirm the compressed RAW data as an image, each color channel image is decoded by JPEG, JPEG-LS, JPEG-2000, etc., and the decoded color channels are synthesized. Thus, it is necessary to reconstruct a color mosaic image and convert the reconstructed color mosaic image into a full color image by a demosaicing process. Therefore, there is a problem that it is not possible to confirm RAW data compressed using an existing decoder as an image.

  However, in order to efficiently save the RAW data, it is very important that the RAW data can be compressed with high compression and low distortion, and that the compressed RAW data can be easily confirmed as an image.

  Here, simply confirming the compressed RAW data as an image using an existing decoder is referred to as “preview” or “preview function”, and an image generated for preview is “ This will be referred to as a “preview image”.

  As described above, the existing RAW data compression method as described in Non-Patent Documents 6, 7, 8, and 9 does not provide a preview function, and only confirms the compressed RAW data as an image. Even in this case, it is necessary to perform complicated image decoding processing, color mosaic image reconstruction, and demosaicing processing.

  Furthermore, the ZIP format and the RAR format are also known as electronic data compression methods. However, even when the RAW data is compressed in the ZIP format or the RAR format, the RAW data can be simply compressed using an existing decoder. There is a problem that data cannot be confirmed as an image.

  The present invention has been made under the circumstances as described above, and an object of the present invention is to provide a RAW data compression method for directly compressing RAW data based on JPEG which is most widely used as an image compression method. Specifically, RAW data acquired by a single-plate solid-state imaging device having a color filter array (CFA) is compressed with high compression and low distortion, and the compressed RAW data is easily confirmed as an image with a JPEG decoder. The object is to provide a RAW data compression method capable of realizing a preview function.

  The present invention relates to a RAW data compression method for directly compressing RAW data observed by a single-plate solid-state imaging device having a color filter array (CFA) as “JPEG data capable of decoding RAW data”. Formulates the decoding process and downsampling of the JPEG decoder, and the JPEG data corresponding to the quantized DCT coefficient that reduces the error between the RAW data to be decoded and the observed RAW data, The “RAW data can be decoded by generating it as“ decoded JPEG data ”or by performing an iterative operation to obtain the quantized DCT coefficient in consideration of the influence of quantization of the DCT coefficient. JPEG data "or by using the redundancy of the number of parameters By reducing the number of parameters of the DCT coefficient, the influence of the quantization error of the DCT coefficient on the RAW data becomes equal regardless of the elements by generating the “JPEG data capable of decoding RAW data”. By redesigning the quantization table as described above, it is effectively achieved by generating the “JPEG data capable of decoding RAW data”.

  In addition, the object of the present invention is to perform an iterative operation to obtain the quantized DCT coefficient in consideration of the influence of quantization of the DCT coefficient, and to reduce the number of parameters of the DCT coefficient using the redundancy of the number of parameters. And re-designing the quantization table so that the influence of the quantization error of the DCT coefficient on the RAW data is the same regardless of the elements, thereby generating the “JPEG data capable of decoding the RAW data”. Is achieved more effectively.

  The RAW data compression method according to the present invention is a method of directly compressing RAW data observed (acquired) by a single-plate solid-state imaging device having a CFA as “JPEG data capable of decoding RAW data” based on JPEG. That is, this is a method of generating “JPEG data capable of decoding RAW data” from the observed RAW data.

  That is, in the present invention, the decoding process and downsampling of the JPEG decoder are formulated, and “JPEG data capable of decoding RAW data is reduced so that an error between the decoded RAW data and the actually observed RAW data is reduced. Is generated.

  According to the present invention, “JPEG data capable of decoding RAW data” is generated from actually observed RAW data, and the RAW data (actually observed RAW data) is compressed with high compression and low distortion. It has an excellent effect that it can be previewed easily.

  Further, in the present invention, the actually observed RAW data is directly compressed as “data that can be decoded by a JPEG decoder”, that is, “JPEG data that can be decoded by RAW data”. For example, when previewing, there is a merit that it is not necessary to make full color by demosaicing processing.

Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

<1> Outline of the present invention The present invention compresses RAW data acquired by a single-plate solid-state imaging device having a color filter array (CFA) with high compression and low distortion, and the compressed RAW data is a JPEG decoder. The present invention relates to a RAW data compression method capable of realizing a preview function for easily confirming an image as an image.

That is, the RAW data compression method of the present invention is a method of directly compressing RAW data observed (acquired) by a single-plate solid-state imaging device having a CFA as “JPEG data capable of decoding RAW data” based on JPEG. Yes, that is, a method of generating “JPEG data capable of decoding RAW data” from observed RAW data.

<2> Outline of JPEG compression Here, JPEG compression will be briefly described.
<2-1> JPEG encoding JPEG compression is widely known as a compression method for full-color images. FIG. 1 shows an outline of JPEG encoding.

  As shown in FIG. 1, first, “RAW data” observed by a single-plate color imaging device having a color filter array (CFA) is demosaiced and converted into a full-color image (RGB color image).

  Next, this full-color image (RGB color image) is converted into a YCrCb color space, subjected to discrete cosine transform (DCT), and then subjected to quantization processing. The data that has been subjected to the discrete cosine transform thus quantized is called “quantized DCT coefficient”.

Finally, the quantized DCT coefficients are run-length and Huffman encoded to generate JPEG data. Color differences (Cr, Cb) are often thinned out in the spatial direction. Although the present invention also supports color difference thinning, for the sake of simplicity, the following description does not consider color difference thinning. Encoded for each minimum encoding unit (MCU). When the color difference thinning is not performed, the MCU has a size of 8 × 8.

<2-2> Formulation of JPEG Decoding FIG. 2 shows an outline of a method for decoding JPEG data. Since decoding is also performed for each minimum coding unit (MCU), only decoding of one MCU is considered here.

に変換される。ＪＰＥＧデータは、量子化ＤＣＴ係数

と一対一に対応している。 As shown in FIG. 2, first, JPEG data is subjected to run length decoding and Huffman decoding, and quantized DCT coefficients.

Is converted to JPEG data is quantized DCT coefficient

And one-to-one correspondence.

は、逆量子化処理、逆離散コサイン変換、および色変換が施され、フルカラー画像（ＲＧＢカラー画像）が復号される。それぞれの処理は、行列演算で表現することができ、すなわち、量子化ＤＣＴ係数

の復号過程は、数１のように表される。 Next, quantized DCT coefficients

Are subjected to inverse quantization processing, inverse discrete cosine transform, and color transform, and a full color image (RGB color image) is decoded. Each process can be expressed by a matrix operation, that is, a quantized DCT coefficient.

The decoding process is expressed as follows.

ここで、ベクトル

はフルカラー画像（ＲＧＢカラー画像）のベクトル表現を表す。また、行列

は逆量子化を、行列

は逆離散コサイン変換を、行列

は色変換を、それぞれ表す行列である。

Where vector

Represents a vector representation of a full color image (RGB color image). Matrix

Is the inverse quantization, matrix

Is the inverse discrete cosine transform, matrix

Is a matrix representing each color conversion.

及びフルカラー画像

は１９２次元のベクトルであり、行列

は１９２×１９２次元の行列である。 Now consider MCU 8x8, so quantized DCT coefficients

And full color image

Is a 192-dimensional vector, matrix

Is a 192 × 192 dimensional matrix.

Further, RAW data can be extracted (decoded) from the decoded full-color image (RGB color image) by a simple operation such as downsampling.

<3> JPEG Data that can Decode RAW Data Here, the characteristics required for RAW data compression will be described first. Then, a concept such as “JPEG data capable of decoding RAW data”, which is a feature of the present invention, is proposed and its features are described.

<3-1> Features Required for RAW Data Compression Consider irreversible compression of RAW data. At this time, the characteristics required for RAW data compression are, first, that RAW data observed by a single-plate color image sensor having a CFA can be decoded with a small error with as little information as possible. If these properties (features) are satisfied, the compression efficiency is high.

  It is also important that RAW data can be easily confirmed as an image. The function of confirming the compressed RAW data as an image is called a preview. Although the preview image does not require high image quality, it needs to be easily imaged by a decoder that has already been widely used. This preview function is considered to be a very important and indispensable function when considering the confirmation of images on the digital camera and the search for images.

  By the way, at present, RAW data observed by a single-plate color image sensor having a CFA is generally processed according to the flow shown in FIG. As can be seen from FIG. 3, it is not considered that RAW data, which is the output of a single-plate color image sensor, is stored with high compression efficiency, and compression only compresses and stores a demosaic-processed color image. Only that is taken into account.

  Further, the currently used storage format for RAW data is non-compressed, and an image for preview cannot be generated by an existing decoder. Further, the preview is not taken into account in the existing RAW data compression method. The existing RAW data compression method is basically based on spatial separation of RAW data into each color channel as shown in the processing flow of FIG. 4 (Non-Patent Documents 6, 7, 8 and 9 are used). reference). For this reason, a preview image cannot be generated easily.

  When only the compression efficiency is considered, it can be said that the method of compressing RAW data using the ZIP format or the RAR format, which is lossless compression, is one of the compression methods with high compression efficiency. It is considered that irreversible compression is also possible by using the ZIP format or the RAR format after irreversible preprocessing so as to increase the compression efficiency. However, such a compression method cannot naturally easily generate a preview image.

Also, considering the widespread use of compression formats, it is extremely important for practical use to be able to be decoded by a currently widely used decoder. Therefore, the present invention proposes “JPEG data capable of decoding RAW data” as follows.

<3-2> Features of “JPEG data capable of decoding RAW data” A full-color image can be obtained by decoding certain JPEG data. RAW data is extracted by down-sampling the obtained full-color image. The operation for extracting RAW data from a full-color image can be designed arbitrarily, but here, downsampling is considered for simplicity.

  When the RAW data extracted in this way has a small error with respect to the observed RAW data, the original JPEG data is referred to as “JPEG data capable of decoding RAW data”.

  Such “JPEG data capable of restoring RAW data” has a feature that a preview image is generated by simple decoding by a JPEG decoder. Further, the RAW data is decoded by down-sampling the preview image.

  Next, the characteristics of “JPEG data capable of decoding RAW data” are formulated.

を利用して、数２のように表される。 First, let us consider decoding certain JPEG data and generating (decoding) RAW data by downsampling a full color image obtained by decoding. If the generated (decoded) RAW data matches the observed RAW data, the JPEG data used to generate (decode) the RAW data is JPEG data that can restore the RAW data. I can say that. This condition is a matrix of number 1

And is expressed as in Equation 2.

ここで、ベクトル

は観測されたＲＡＷデータを表すベクトルを、行列

はフルカラー画像からＲＡＷデータを生成（復号）するためのダウンサンプリングを表す行列を、行列

は量子化ＤＣＴ係数

からＲＡＷデータを復号するための復号行列を、それぞれ表す。

Where vector

Is a vector representing the observed raw data, a matrix

Is a matrix representing downsampling for generating (decoding) RAW data from a full-color image.

Is the quantized DCT coefficient

1 represents a decoding matrix for decoding RAW data from each.

  If Equation 2 is satisfied, the RAW data observed without error can be decoded from the JPEG data. If such JPEG data can be encoded, it can be considered that the RAW data is reversibly compressed.

  This concept of lossless compression of RAW data can be extended to lossy compression of RAW data. The irreversible compression of RAW data is to encode JPEG data such that the error between the observed RAW data and the decoded RAW data expressed by Equation 3 is sufficiently small.

ここで、‖・‖_２はＬ２ノルムを表す。

Here, ‖ · ‖ ₂ represents the L2 norm.

に対応するＪＰＥＧデータを、「ＲＡＷデータ復号可能なＪＰＥＧデータ」と定義する。 In the present invention, a quantized DCT coefficient that reduces the error E (ie, the error between the observed RAW data and the decoded RAW data) expressed by Equation (3).

The JPEG data corresponding to is defined as “RAW data decodable JPEG data”.

  The “RAW data decodable JPEG data” defined as described above can not only decode RAW data but also a full color image by a normal JPEG decoder.

を生成すれば、「ＲＡＷデータ復号可能なＪＰＥＧデータ」が得られる訳である。

＜４＞「ＲＡＷデータを復号可能なＪＰＥＧデータ」の生成方法
＜３−２＞では、「ＲＡＷデータを復号可能なＪＰＥＧデータ」の特徴を述べた。 Therefore, in the RAW data compression method of the present invention, the quantized DCT coefficient that reduces the error between the decoded RAW data and the observed RAW data.

Is generated, “RAW data decodable JPEG data” is obtained.

<4> Method of Generating “RAW Data Decodable JPEG Data” In <3-2>, the characteristics of “RAW data decodable JPEG data” are described.

を求める方法を具体的に述べる。 Here, a method of generating “JPEG data capable of decoding RAW data” from the observed RAW data will be specifically described. That is, a quantized DCT coefficient that reduces an error between decoded RAW data and observed RAW data.

The method for obtaining the value will be specifically described.

  First, a case where DCT coefficients are continuous will be described. Next, an iterative calculation method for obtaining the quantized DCT coefficient will be described. Further, a method for reducing the number of parameters of the DCT coefficient and a method for redesigning the quantization table will be described.

  That is, in the RAW data compression method of the present invention, in order to generate “JPEG data that can decode RAW data” from the observed RAW data, the decoding process and downsampling of the JPEG decoder are formulated and decoded. Based on the idea of generating “JPEG data capable of decoding RAW data” so as to reduce the error between the RAW data and the observed RAW data, the quantized DCT is considered in consideration of the quantization effect of the DCT coefficient. It is possible to compress RAW data with high compression and low distortion by performing iterative calculation to obtain coefficients, reducing the number of parameters using the redundancy of the number of parameters, and redesigning the quantization table. .

Since the quantized DCT coefficient and the JPEG data have a one-to-one correspondence with run length and Huffman coding, a method for generating a discrete DCT coefficient will be described here.

<4-1> In the case of continuous DCT coefficient It is considered to minimize the error E expressed by Equation 3 for each 8 × 8 MCU. Here, first, continuous DCT coefficients are considered without considering DCT coefficient quantization.

は６４次元で、求めるべきフルカラー画像のＤＣＴ係数であるベクトル

は１９２次元で、復号化行列（復号行列）

は１９２×６４次元である。 First, check the main matrix and vector dimensions. Vector that is the observed RAW data

Is a 64D vector which is the DCT coefficient of the full color image to be obtained

Is a 192 dimension, decoding matrix (decoding matrix)

Is 192 × 64 dimensions.

は無数にある。 That is, since the number of unknowns to be obtained (192) is larger than the number (64) of the constraint equation, the continuous DCT coefficient that makes the error E expressed by Equation 3 zero.

Is innumerable.

  If the quantization of the DCT coefficient is not taken into consideration, one of the DCT coefficients that makes the error E expressed by Equation 3 zero is given as a least squares minimum norm solution of Equation 2.

  This least square minimum norm solution is uniquely obtained as shown in Equation 4 using the Moore-Penrose generalized inverse matrix.

ここで、行列

は復号行列

のMoore-Penroseの一般化逆行列を表す。

Where the matrix

Is the decoding matrix

Represents the generalized inverse of Moore-Penrose.

を量子化する必要がある。連続ＤＣＴ係数

の量子化に伴い、量子化誤差も同時に発生してしまう。

＜４−２＞量子化ＤＣＴ係数を求めるための繰り返し計算
＜４−１＞で述べたように、一般化逆行列を利用して、ＤＣＴ係数を求めることができる。しかしながら、「ＲＡＷデータを復号可能なＪＰＥＧデータ」を生成するためには、ＤＣＴ係数の量子化を行わなければならず、よって量子化誤差が発生する。 However, in order to generate “JPEG data that can decode RAW data”, continuous DCT coefficients

Needs to be quantized. Continuous DCT coefficient

Quantization errors also occur at the same time.

<4-2> Iterative Calculation for Obtaining Quantized DCT Coefficient As described in <4-1>, a DCT coefficient can be obtained using a generalized inverse matrix. However, in order to generate “JPEG data capable of decoding RAW data”, the DCT coefficients must be quantized, which causes a quantization error.

  In order to reduce this quantization error, a method for obtaining the quantized DCT coefficient by iterative calculation is proposed here.

FIG. 5 shows an iterative calculation procedure for obtaining quantized DCT coefficients. As shown in FIG. 5, the quantization DCT coefficient is updated so as to cancel the quantization error. Thus, it is possible to reduce the quantization error by performing repeated calculation.

<4-3> Reduction in number of parameters of DCT coefficient The number of DCT coefficients to be obtained (unknown number) is 192, and the number of observed RAW data (number of constraint equations) is 64. As described above, the error E represented by Equation 3 can be made zero without considering the conversion.

  That is, it is not necessary to use all DCT coefficients. Therefore, a decoding process such as Equation 5 is considered.

ここで、行列

は対角要素が

である対角行列を、ベクトル

は要素が０又は１のベクトルをそれぞれ表す。なお、ベクトル

の次元は、ベクトル

の次元と同じ１９２である。

Where the matrix

Is the diagonal element

A diagonal matrix that is a vector

Represents vectors with 0 or 1 elements, respectively. Vector

Dimension is a vector

The same dimension as 192.

は、数６に表される。 Mathematical 5 least squares minimum norm

Is expressed in Equation 6.

さて、ベクトル

のある要素が０であれば、最小二乗最小ノルム解

の対応する要素は０になる。また、行列

のランクがベクトル

と等しければ、数３で表す誤差Ｅをゼロにする連続ＤＣＴ係数を求めることができる。

Well, vector

If any element of is zero, the least-squares-norm solution

The corresponding element of becomes zero. Matrix

Rank is vector

If so, a continuous DCT coefficient that makes the error E expressed by Equation 3 zero can be obtained.

を設計すればよい。 By the way, the run length and Huffman coding of the quantized DCT coefficient yields JPEG data. Therefore, in order to achieve high compression efficiency, the vector must contain many consecutive zeros.

Should be designed.

を利用した。このようなベクトル

を利用することにより、図６中で１に対応する周波数のＤＣＴ係数のみが利用されることになる。このような方法により、実質的に利用するパラメータ数を１９６から６７へ削減した。また、このとき、行列

のランクがベクトル

と等しい。

＜４−４＞ＤＣＴ係数の量子化テーブルの再設計
国際電気通信連合（ＩＴＵ）は、人の視覚に基づいたＤＣＴ係数のための量子化テーブルを勧告している。この量子化テーブルは広く利用されており、この量子化テーブルを定数倍することによって、圧縮率が変更されている。 In JPEG, zigzag scanning is employed when storing DCT coefficients, and the high-frequency component of the color difference is less perceptible to human vision.

Was used. Vector like this

By using, only the DCT coefficient of the frequency corresponding to 1 in FIG. 6 is used. By such a method, the number of parameters substantially used was reduced from 196 to 67. At this time, the matrix

Rank is vector

Is equal to

<4-4> Redesign of DCT Coefficient Quantization Table The International Telecommunication Union (ITU) recommends a quantization table for DCT coefficients based on human vision. This quantization table is widely used, and the compression ratio is changed by multiplying this quantization table by a constant number.

  By the way, the quantization is performed on the DCT coefficient, and the quantization error of the DCT coefficient propagates to the quantization error of the RAW data. Further, the quantization error is generated in the same manner without depending on the element of the DCT coefficient.

  Therefore, in the present invention, the quantization table is redesigned so that the influence of the quantization error of the DCT coefficient on the RAW data becomes equal regardless of the elements.

  First, a quantization error in RAW data is obtained from a decoding process from continuous DCT coefficients to RAW data. The decoding process from continuous DCT coefficients to RAW data can be modified as shown in Equation 7.

ここで、下記数８が成立する。

Here, the following equation 8 holds.

また、ベクトル

は行列

のｉ番目の列ベクトルを、ｆ_ｉはベクトル

のｉ番目の要素を、行列

は逆量子化を表す行列を、ｑ_ｉはｉ番目の要素の量子化間隔を、ｅ_ｉはｉ番目のＤＣＴ係数の量子化誤差を、ｎはベクトル

の要素数を、それぞれ表す。

Vector

Is a matrix

The i th column vector, f _i is the vector

The i th element of the matrix

Is a matrix representing inverse quantization, q _i is the quantization interval of the i th element, e _i is the quantization error of the i th DCT coefficient, and n is a vector

Represents the number of elements.

はパラメータ削減を考慮した形式になる。このとき、ＲＡＷデータにおける量子化誤差

は、数９のように表される。 When the method for reducing the number of DCT coefficient parameters described in <4-3> is used, the matrix

Takes the form of parameter reduction. At this time, the quantization error in the RAW data

Is expressed as in Equation 9.

各ＤＣＴ係数の要素の量子化誤差ｅ_ｉが、ＲＡＷデータに与える影響を等しくするために、各要素の量子化間隔ｑ_ｉを数１０のように設計する。

In order to equalize the influence of the quantization error e _i of each DCT coefficient element on the RAW data, the quantization interval q _i of each element is designed as shown in Equation 10.

ここで、ｚは圧縮率を調節するための定数である。

なお、本発明では、＜４−２＞で述べた「量子化ＤＣＴ係数を求めるための繰り返し計算方法」、＜４−３＞で述べた「ＤＣＴ係数のパラメータ数の削減方法」、＜４−４＞で述べた「ＤＣＴ係数の量子化テーブルの再設計方法」をそれぞれ組合せが可能であり、最も効果がある方法は、３種類の方法全てを組み合わせたものである。

Here, z is a constant for adjusting the compression rate.

In the present invention, “an iterative calculation method for obtaining a quantized DCT coefficient” described in <4-2>, “a method for reducing the number of parameters of a DCT coefficient” described in <4-3>, <4- The “method for redesigning the DCT coefficient quantization table” described in 4> can be combined, and the most effective method is a combination of all three methods.

  In the above-described embodiment of the present invention, the generation method of “JPEG data capable of decoding RAW data” has been described. However, the present invention is not limited to this, and for example, not only JPEG but also JPEG-LS and JPEG-2000. The technical idea of the present invention can also be applied to other image compression formats.

In the embodiment of the present invention described above, “JPEG data that can restore RAW data” is simply decoded by a JPEG decoder to generate a preview image, and by downsampling the preview image, Although RAW data is described as being decoded, the present invention is not limited to downsampling, and other simple operations can use, for example, a method that uses the average of neighboring pixels.

<5> Experiment Here, the RAW data compression method of the present invention is applied to the standard image and the actual image, and compared with the conventional method, the effectiveness of the present invention is confirmed.

  That is, “repetitive calculation for obtaining quantized DCT coefficients” described in <4-2>, “reduction in the number of parameters of DCT coefficients” described in <4-3>, and “4-4> described in“ 4-4> ” The effect of “Redesign of DCT coefficient quantization table” is confirmed.

  Specifically, the number of repetitions of the iterative calculation is fixed to 3 times, and the four types of RAW data compression methods according to the present invention are used when the parameter number reduction and the quantization table redesign are used and not used, respectively. It was investigated.

  The original image (standard image) shown in FIG. 7 was downsampled to generate RAW data corresponding to the Bayer color filter array. With respect to this RAW data (RAW data of the original image), “JPEG data capable of decoding RAW data” was generated by using the above-described four RAW data compression methods according to the present invention. FIG. 8 shows the result of obtaining the compression rate and the CPSNR for the RAW data while changing the quantization interval.

  In FIG. 8, the compression ratio on the horizontal axis is the ratio between the size of “JPEG data that can decode RAW data” generated by the present invention and the data size of RAW data (RAW data of the original image). A larger value indicates higher compression. Further, the CPSNR of the RAW data on the vertical axis represents the mean square error between the decoded RAW data and the RAW data of the original image in decibels. The larger the value, the closer to the original image. Yes.

  That is, in the graph of FIG. 8, the more data exists in the upper right, the more RAW data is compressed at a high compression rate and low distortion. FIG. 8 confirms that the RAW data compression method that simultaneously reduces the number of parameters and redesigns the quantization table compresses the RAW data most efficiently.

  FIG. 9 shows a preview image of “JPEG data capable of decoding RAW data” generated by the above-described four types of RAW data compression methods according to the present invention when the compression rate is about 4. The data can be confirmed as an image sufficiently from the preview image shown in FIG. Furthermore, it can be seen from FIG. 9 that among the preview images according to the four types of RAW data compression methods, the image quality of FIG. 9D using the reduction in the number of parameters and the redesign of the quantization table at the same time is the best.

  Next, the RAW data compression method of the present invention that simultaneously utilizes the reduction in the number of parameters and the redesign of the quantization table described above is performed using the existing JPEG, JPEG-LS, and JPEG-2000 methods. Compare with the RAW data compression method.

  First, RAW data obtained by down-sampling the original image shown in FIG. 7 is demosaiced by the method of Adams et al. Described in Non-Patent Document 10 to obtain a full-color image. JPEG data, JPEG-LS data, and JPEG-2000 data were respectively generated for the obtained full color image while changing the quantization interval. RAW data is obtained by decoding and down-sampling each image data.

  FIG. 10 shows a comparison between the RAW data compression method of the present invention and an existing RAW data compression method using JPEG, JPEG-LS and JPEG-2000. FIG. 10 confirms that the CPSNR of the RAW data compression method of the present invention is the highest value at any compression rate.

  A similar comparison was performed on the actually captured image (actual image) shown in FIG. Note that a camera having a single-plate color image sensor is used to capture the actual image in FIG. Therefore, FIG. 11 shows a full color image.

  When compressing by the conventional method, various compression methods were applied to the full-color image obtained by the demosaicing process by the method of Adams et al. FIG. 12 shows a comparison between the RAW data compression method of the present invention and the existing RAW data compression method for the actual image shown in FIG.

  As can be seen from FIG. 10 and FIG. 12, although the tendency differs somewhat depending on the target image, also in FIG. 12, the RAW data compression method of the present invention shows the highest CPSNR at any compression rate.

  As can be seen from the experimental results described above, it was confirmed that the RAW data compression method of the present invention can efficiently compress RAW data and can easily generate a preview image.

  Here, as an application method of the present invention, on a digital camera using a single-plate color image sensor having CFA, first, RAW data (that is, observed RAW data) output from the single-plate color image sensor is recorded. Compressed and stored by the RAW data compression method of the invention. After that, the compressed RAW data stored on the digital camera is transferred to an information processing device such as a personal computer, and then the data transferred in the information processing device such as a personal computer takes a long time to calculate but is highly accurate demosaicing. An application of processing is conceivable.

  Further, the “JPEG data capable of decoding RAW data” generated by the present invention can be used for super-resolution processing.

  FIG. 13 shows various compression results for the actual image of FIG. FIG. 13 (A) shows the result of compressing the demosaicing processing by the method of Adams et al. Described in Non-Patent Document 10 with JPEG-2000, and FIG. 13 (B) shows the RAW data compressed by the present invention. FIG. 13C shows the result of demosaicing processing of the resulting preview image by the method of Hirakawa et al. Described in Non-Patent Document 11 based on the RAW data compressed in the present invention.

  Although Hirakawa et al.'S demosaicing process described in Non-Patent Document 11 is known as a method capable of demosaicing with higher definition than the conventional method, it is calculated in comparison with the method of Adams et al. Described in Non-Patent Document 10. Cost is high.

That is, FIG. 13B shows an image for confirmation on a digital camera obtained by applying the present invention, and FIG. 13C shows an image finally obtained by applying the present invention. Conceivable. By comparing FIG. 13A and FIG. 13C, it was confirmed that the image of FIG. 13C to which the present invention was applied was reproduced in detail.

As described above, in the present invention, RAW data compression in which RAW data observed (acquired) by a single-plate solid-state imaging device having CFA is directly compressed as “JPEG data capable of decoding RAW data” based on JPEG. Developed a method.

  FIG. 14 shows the flow of image processing when the RAW data compression method according to the present invention is applied to a digital camera using a single-plate color image sensor having a CFA. Since the RAW data compression method of the present invention can also be considered as a “RAW data encoding method”, the present invention is expressed as the encoder of the present invention in FIG.

  As shown in FIG. 14, first, the observed RAW data is encoded and stored by the encoder of the present invention, that is, the observed RAW data is “decoded RAW data by the RAW data compression method of the present invention. It is possible to compress and save “JPEG data”, decode the compressed “RAW data JPEG data that can be decoded” using an existing JPEG decoder, and use the decoded full-color image as a preview image. Also, by down-sampling the decoded full-color image and further performing demosaicing processing, a highly accurate full-color image can be obtained.

It is a schematic diagram which shows the outline | summary of JPEG encoding. It is a schematic diagram which shows the outline | summary of the method of decoding JPEG data. It is a schematic diagram which shows the process of the RAW data observed with the single plate type color image sensor which has CFA generally performed now. It is a schematic diagram which shows the flow of a process of the existing RAW data compression method. It is a schematic diagram showing the flow of image processing when the RAW data compression method according to the present invention is applied to a digital camera using a single-plate color image sensor having a CFA. In this invention, it is a schematic diagram for demonstrating the procedure of iterative calculation for calculating | requiring a quantization DCT coefficient. It is a schematic diagram for demonstrating the DCT coefficient utilized by JPEG. It is a figure which shows the original image used in the experiment conducted in order to confirm the effectiveness of this invention. It is a graph which shows the relationship between a compression rate and CPSNR with respect to RAW data, when four types of RAW data compression methods based on this invention are applied. It is a figure which shows the preview image of "JPEG data which can decode RAW data" produced | generated by four types of RAW data compression methods based on this invention. It is a figure which shows the comparison of the RAW data compression method of this invention, and the existing RAW data compression method using JPEG, JPEG-LS, and JPEG-2000. It is a figure which shows the real image used in the experiment conducted in order to confirm the effectiveness of this invention. It is a figure which shows the comparison of the RAW data compression method of this invention, and the existing RAW data compression method with respect to the real image of FIG. It is a figure which shows the various compression results with respect to the real image of FIG.

Claims (5)

A RAW data compression method for directly compressing RAW data observed by a single-plate solid-state imaging device having a color filter array (CFA) as “JPEG data capable of decoding RAW data”,
Formulating the decoding process and downsampling of the JPEG decoder, the JPEG data corresponding to the quantized DCT coefficient that reduces the error between the decoded RAW data and the observed RAW data can be converted into the “RAW data can be decoded. RAW data compression method, wherein the RAW data compression method is generated.   2. The RAW data according to claim 1, wherein the “RAW data can be decoded” is generated by performing an iterative operation to obtain the quantized DCT coefficient in consideration of the quantization effect of the DCT coefficient. Compression method.   2. The RAW data compression method according to claim 1, wherein the “JPEG data capable of decoding RAW data” is generated by reducing the number of parameters of a DCT coefficient using redundancy of the number of parameters.   2. The “JPEG data capable of decoding RAW data” is generated by redesigning a quantization table so that the influence of quantization errors of DCT coefficients on RAW data becomes equal regardless of factors. RAW data compression method as described.   Considering the influence of the quantization of the DCT coefficient, iterative calculation is performed to obtain the quantized DCT coefficient, the number of parameters of the DCT coefficient is reduced using the redundancy of the number of parameters, and the quantization of the DCT coefficient is performed. 2. The RAW data compression according to claim 1, wherein the “JPEG data capable of decoding RAW data” is generated by redesigning a quantization table so that an influence of an error on RAW data is equal regardless of elements. Method.

Images