JP3864748B2 - Image processing apparatus, electronic camera, and image processing program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus that can switch between a normal compression mode and a RAW compression mode.
The present invention relates to an electronic camera equipped with this image processing apparatus.
The present invention relates to an image processing program for realizing the image processing apparatus by a computer.
[0002]
[Prior art]
Conventional electronic cameras are known that can select the following two types of compression modes when recording image data.
[0003]
(1) Normal compression mode: A mode in which lossy compression such as JPEG is performed after signal processing such as color interpolation is performed on the output of the image sensor. Note that the general normal compression mode is often subdivided into finer compression modes depending on the compression size setting of the lossy compression (fine, basic, etc.).
[0004]
(2) RAW compression mode: A mode in which lossless compression such as variable-length coding (including pseudo lossless compression that can cause minute compression distortion) is performed on data before color interpolation (so-called RAW data). .
[0005]
In these two compression modes, the processing procedure and processing contents of the compression processing are greatly different. Therefore, the conventional electronic camera is provided with two compression processing units in preparation for these two compression modes.
[0006]
For example, the compression processing in the normal compression mode is often left to a JPEG compression IC with a hardware configuration. In addition, the compression processing in the RAW compression mode is often left to software compression processing by an MPU (microprocessor unit) inside the electronic camera.
[0007]
[Problems to be solved by the invention]
As described above, the conventional electronic camera includes a dedicated compression mechanism for each of the normal compression mode / RAW compression mode. For this reason, there has been a problem that the configuration of the electronic camera is complicated and that the switching of the signal path at the time of mode switching is complicated. The present invention has been made in view of such problems, and an object thereof is to provide a configuration for easily switching between a normal compression mode and a RAW compression mode.
[0008]
[Means for Solving the Problems]
The present invention will be described below.
[0009]
<Claim 1>
An image processing apparatus according to a first aspect of the present invention is an image processing apparatus for processing “RAW data in which a plurality of types of signal components constituting a color image are mixedly arranged on a pixel array”, and an instruction from the outside A mode setting unit that switches between normal compression mode and RAW compression mode according to the input, and interpolation processing of RAW data to compensate for missing signal components in the pixel unit in the normal compression mode, and bypass interpolation processing in the RAW compression mode And a signal processing unit that compresses an output of the signal processing unit. In particular, the image compression unit includes a frequency conversion unit that performs frequency conversion on the output of the signal processing unit and outputs a conversion coefficient, and an encoding unit that encodes the conversion coefficient. In this image processing apparatus, the operation in the RAW compression mode is realized by bypassing at least the interpolation processing. Further, in the RAW compression mode, the image compression unit adds a code indicating that no interpolation processing is performed on the compressed image data.
[0010]
<Claim 2>
The image processing apparatus according to a second aspect of the present invention is the image processing apparatus according to the first aspect, wherein the signal processing unit classifies and outputs the RAW data according to the type of the signal component in the RAW compression mode.
<Claim 3>
According to a third aspect of the present invention, there is provided the image processing device according to the first aspect, wherein the signal processing unit classifies the RAW data according to the type of the signal component in the RAW compression mode, and includes a G1 component, a G2 component, Output as R component and B component. On the other hand, the image compression unit compresses the image for each of the G1, G2, R, and B components in the RAW compression mode.
[0011]
<Claim 4>
An image processing apparatus according to a fourth aspect of the present invention is the image processing apparatus according to any one of the first to third aspects, wherein the signal processing unit performs color coordinate conversion on the image signal in the normal compression mode. In the RAW compression mode, this color coordinate conversion is bypassed.
[0012]
<Claim 5>
An image processing apparatus according to a fifth aspect of the present invention is the image processing apparatus according to any one of the first to fourth aspects, wherein the image compression unit compresses the image in the RAW compression mode as compared with that in the normal compression mode. Reduce compression distortion.
[0013]
<Claim 6>
An image processing apparatus according to a sixth aspect of the present invention is the image processing apparatus according to any one of the first to fourth aspects, wherein when the image compression unit is in the normal compression mode, the file size after compression is set as a target. In the RAW compression mode, the compression distortion is adjusted within the target range.
[0015]
< Claim 7 >
An electronic camera according to a seventh aspect of the present invention is an image pickup section that picks up a subject image and generates RAW data in which a plurality of types of signal components constituting a color image are mixedly arranged on a pixel array, and Or an image processing apparatus according to any one of claims 6 to 6 . In this electronic camera, RAW data generated by the imaging unit is processed by an image processing apparatus.
[0016]
[ Claim 8]
An image processing program according to an eighth aspect of the present invention causes a computer to function as the mode setting unit, the signal processing unit, and the image compression unit according to any one of the first to sixth aspects. .
[0017]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments according to the present invention will be described below with reference to the drawings.
[0018]
<< Configuration of this embodiment >>
FIG. 1 is a diagram illustrating an electronic camera 11 (including an image processing apparatus) according to the present embodiment.
In FIG. 1, a photographing lens 12 is attached to the electronic camera 11. In the image space of the photographic lens 12, the imaging surface of the imaging element 13 is arranged. A Bayer array color filter array is disposed on the imaging surface of the imaging element 13.
[0019]
RAW data as shown in FIG. 2 is output from the image sensor 13. This RAW data is a mixture of a plurality of types of signal components (in this case, RGB) constituting a color image on a pixel array. Here, since the single-plate image sensor 13 is used, one type of signal component is allocated to each RAW data.
Thus, the RAW data output from the image sensor 13 is digitized via the A / D conversion unit 14. The digitized RAW data is processed through the signal processing unit 15 and the image compression unit 16 and then recorded as a compressed file in the recording unit 17.
[0020]
The electronic camera 11 is provided with a mode setting unit 18. The mode setting unit 18 receives an instruction input (in this case, a setting operation from a user) via an operation member (not shown), and switches the mode setting of the compression mode according to the instruction input. Next, the internal data flow of the signal processing unit 15 shown in FIG. 1 will be described. In the signal processing unit 15, a path switching unit 19, an interpolation processing unit 20, a color coordinate conversion unit 21, and a pixel rearrangement processing unit 22 are provided. Among these, the path switching unit 19 switches the signal path of the RAW data in accordance with the compression mode switching of the mode setting unit 18.
[0021]
That is, in the “normal compression mode”, the RAW data is processed by the interpolation processing unit 20 and the color coordinate conversion unit 21 and then output to the image compression unit 16.
On the other hand, in the “RAW compression mode”, the RAW data is processed by the pixel rearrangement processing unit 22 after bypassing the interpolation processing unit 20 and the color coordinate conversion unit 21. After such processing, the RAW data is output to the image compression unit 16.
[0022]
<< Correspondence with Invention >>
The correspondence relationship between the invention and this embodiment will be described below. Note that the correspondence relationship here illustrates one interpretation for reference, and does not limit the present invention.
The mode setting unit described in the claims corresponds to the mode setting unit 18.
The signal processing unit described in the claims corresponds to the signal processing unit 15.
The image compression unit described in the claims corresponds to the image compression unit 16.
The imaging unit recited in the claims corresponds to the imaging element 13.
[0023]
<< Operation of this embodiment >>
Hereinafter, the operation of the electronic camera 11 will be described.
Here, in order to clarify the correspondence with the invention, first, the operation of the signal processing unit 15 will be described separately for each compression mode. Thereafter, a common processing procedure performed by the image compression unit 16 will be described.
[0024]
[Description of Operation of Signal Processing Unit 15 in Normal Compression Mode]
First, the operation of the signal processing unit 15 set to the normal compression mode will be described. The path switching unit 19 transmits the RAW data input to the signal processing unit 15 to the interpolation processing unit 20 according to the setting of the normal compression mode.
The interpolation processing unit 20 first temporarily stores the transmitted RAW data in a buffer memory (not shown) in the signal processing unit 15. Subsequently, the interpolation processing unit 20 reads RAW data from the buffer memory for each appropriate processing unit, and executes a known interpolation process.
[0025]
As an interpolation process here,
(1) A method of compensating for a missing signal component with a weighted average value or median value of surrounding pixels. (2) An interpolation method for determining the image structure (edge line direction, etc.) of surrounding pixels and preferentially using a direction (similar direction) with a small amount of change to calculate a weighted average value or median value.
(3) A method of performing color difference conversion on a signal component and executing interpolation processing on the color difference space.
Many other interpolation methods can be used.
[0026]
By such an interpolation process, as shown in FIG. 2, the signal component missing in the pixel unit in the RAW data is compensated. The interpolation processing unit 20 transmits the RGB data after the interpolation processing to the color coordinate conversion unit 21.
The color coordinate conversion unit 21 performs color coordinate conversion such as multiplying the transmitted RGB data by a color conversion matrix. By this color coordinate conversion, the RGB data is converted into a component for normal compression such as YCbCr (see FIG. 3). At this time, data reduction such as pixel thinning may be executed for colors and color differences with low visual sensitivity.
The color coordinate conversion unit 21 sequentially outputs each component of YCbCr subjected to the color coordinate conversion to the image compression unit 16.
[0027]
[Description of Operation of Signal Processing Unit 15 in RAW Compression Mode]
Next, the operation of the signal processing unit 15 set to the RAW compression mode will be described.
The path switching unit 19 bypasses the processing of the interpolation processing unit 20 and the color coordinate conversion unit 21 according to the setting of the RAW compression mode, and transmits the RAW data to the pixel rearrangement processing unit 22.
[0028]
The pixel rearrangement processing unit 22 classifies the transmitted RAW data for each type of signal component as shown in FIG. As a result, a plurality of types of signal components constituting the RAW data are classified and arranged into the following four components.
(1) G1 component composed of G signal components in odd columns and odd rows.
(2) G2 component composed of G signal components in even columns and even rows.
(3) R component composed of R signal components.
(4) B component composed of B signal components.
[0029]
G1 and G2 are images in which the sampling positions are slightly shifted by one pixel in the vertical and horizontal directions (that is, in an oblique direction). Therefore, the data correlation between G1 and G2 is generally very high except for special image patterns such as diagonal stripes. Therefore, encoding efficiency may be improved by using “difference between G1 component and G2 component” instead of the G2 component.
Alternatively, one component may be created by interweaving the G1 component and the G2 component in units of columns or rows like a fabric.
The pixel rearrangement processing unit 22 sequentially outputs the components classified and arranged in this manner to the image compression unit 16.
[0030]
[Description of Operation of Image Compression Unit 16]
FIG. 5 is a diagram illustrating a processing procedure of the image compression unit 16.
Hereinafter, the operation of the image compression unit 16 will be described in accordance with the processing procedure shown in FIG.
[0031]
(1) The component size is different between the tiled normal compression mode (Y, Cb, Cr) and the RAW compression mode (R, B, G1, G2). Therefore, in order to efficiently share the subsequent processing procedures, the image compression unit 16 divides each component into predetermined data processing units to obtain a plurality of tiles (rectangular regions). The tiles divided in this way are handled independently in the subsequent processing.
[0032]
(2) Wavelet Transform First, the image compression unit 16 recursively performs discrete wavelet transform (frequency decomposition) for each tile to obtain transform coefficients subjected to subband decomposition.
[0033]
(3) Quantization Next, the image compression unit 16 quantizes the transform coefficient thus obtained with a predetermined quantization width for each subband. This quantization width determines the best limit of compression distortion. Note that in the case of realizing complete lossless compression in the RAW compression mode, the quantization step width is set to “1”.
[0034]
(4) Entropy encoding Here, when a selection region (ROI) to be preferentially encoded is determined, the image compression unit 16 selects a quantized transform coefficient located in the selection region. Shift up by S bits. The S bit is the number of bits determined by the max shift method. In the case of realizing complete lossless compression in the RAW compression mode, this shift-up process becomes meaningless and may be omitted if there is no special circumstance.
Subsequently, the image compression unit 16 classifies the transform coefficients into bits and forms a plurality of bit planes. Note that quantization at the quantization step width “2 to the power of N” may be performed by truncating the lower N planes at this stage.
For these bit planes, the image compression unit 16 executes the following encoding pass in order from the highest-order plane. Note that these encoding passes are defined in JPEG2000.
(1) SP pass (Significance propagation pass)
(2) MR pass (Magnitude refinement pass)
(3) Cleanup pass
By passing through these three encoding passes, the transform coefficient is entropy encoded.
[0035]
(5) Arithmetic Coding Next, the image compression unit 16 performs arithmetic coding on the entropy-encoded encoded data using an MQ coder that is a binary arithmetic code.
[0036]
(6) Bitstream Generation Subsequently, the image compression unit 16 rearranges the arithmetically encoded encoded data according to SNR progressive or other priorities to generate a bitstream.
Here, in the case of the normal compression mode, the image compression unit 16 terminates the bitstream halfway, and adjusts the compressed file size within the target range.
On the other hand, in the case of the RAW compression mode, the image compression unit 16 does not terminate the bit stream and keeps the compression distortion low within the target range. (Note that with perfect lossless compression, the compression distortion is zero.)
[0037]
(7) The header-added image compression unit 16 combines the generated bitstreams into one, adds a file header, and generates a compressed file.
In the case of the RAW compression mode, the image compression unit 16 adds a code indicating that interpolation processing is not performed to the compressed file.
Through such a processing procedure, the compressed file is completed. The recording unit 17 records this compressed file on a recording medium such as a memory card.
[0038]
<< Effects of this embodiment >>
As described above, switching between the normal compression mode and the RAW compression mode is realized by bypassing the interpolation processing unit 20. On the other hand, the image compression unit 16 executes a common processing procedure regardless of the compression mode. Therefore, in the present embodiment, there is almost no need to provide a plurality of image compression units for each compression mode or to switch the signal path in a complicated manner inside the image compression unit 16. As a result, switching between the normal compression mode and the RAW compression mode can be realized very simply with a simple configuration and operation.
[0039]
In the present embodiment, in the RAW compression mode, a plurality of components are generated by classifying the signal components of the RAW data by type. On the other hand, even in the normal compression mode, a plurality of components are generated through interpolation processing and color coordinate conversion. Thus, even when the compression mode is switched, the output of the signal processing unit 15 is common in terms of component output. As described above, by adjusting the component output in the signal processing unit 15, the common processing procedure can be easily performed on the image compression unit 16 side.
[0040]
<< Additional items of embodiment >>
In the above-described embodiment, a case where the present invention is applied to the electronic camera 11 has been described. However, the present invention is not limited to this. For example, the present invention may be realized as a single image processing apparatus.
[0041]
Further, the image processing apparatus of the present invention may be realized by a computer. In this case, the image processing program may be created by converting the operation of the image processing apparatus described above into a program code.
[0042]
In the above-described embodiment, RAW data with a Bayer array is handled. However, the present invention is not limited to this. In general, RAW data is image data before interpolation processing, and refers to data in which a plurality of types of signal components (color, luminance, color difference, etc.) are mixedly arranged on a pixel array. Therefore, in the present invention, it is possible to handle RAW data in general, such as RAW data of a complementary color array, RAW data consisting of a specific color system, and RAW data of a two-plate imaging device.
[0043]
In the above-described embodiment, a JPEG2000-compliant image compression method is employed. However, the present invention is not limited to this. In general, any compression method can be used as long as it can handle lossless compression (including pseudo lossless compression) and lossy compression with a common processing procedure.
[0044]
In the embodiment described above, pixel rearrangement (see FIG. 4) of RAW data is executed in the RAW compression mode. However, the present invention is not limited to this. For example, pixel rearrangement in the RAW compression mode may be omitted.
[0045]
【The invention's effect】
In the present invention, switching of the signal path in the normal compression mode / RAW compression mode is almost completed at the processing stage of the signal processing unit. On the other hand, the image compression unit is unified regardless of the difference between these compression modes.
With such a configuration, there is no need to provide a plurality of image compression units for each compression mode or to switch signal paths in a complicated manner within the image compression unit. As a result, the compression mode can be easily switched with a simple configuration and operation.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating an electronic camera 11 (including an image processing apparatus) according to an embodiment.
FIG. 2 is a diagram for explaining interpolation processing in a normal compression mode.
FIG. 3 is a diagram illustrating color coordinate conversion in a normal compression mode.
FIG. 4 is a diagram for explaining pixel rearrangement in a RAW compression mode.
FIG. 5 is a diagram illustrating a processing procedure of an image compression unit 16;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 11 Electronic camera 12 Shooting lens 13 Image pick-up element 14 A / D conversion part 15 Signal processing part 16 Image compression part 17 Recording part 18 Mode setting part 19 Path switching part 20 Interpolation processing part 21 Color coordinate conversion part 22 Pixel rearrangement processing part

Claims (8)

An image processing apparatus that is provided with RAW data in which a plurality of types of signal components constituting a color image are mixedly arranged on a pixel array and processes the RAW data,
A mode setting unit for switching between normal compression mode / RAW compression mode in response to an external instruction input;
In the normal compression mode, the RAW data is interpolated to compensate for the signal component missing in pixel units, and in the RAW compression mode, a signal processing unit that bypasses the interpolation processing;
An image compression unit that compresses the output of the signal processing unit,
The image compression unit
A frequency converter that converts the output of the signal processing unit and outputs a conversion coefficient;
An encoding unit for encoding the transform coefficient;
An image processing apparatus which realizes the RAW compression mode by at least bypassing the interpolation processing and adds a code indicating that no interpolation processing is performed to the compressed image data in the RAW compression mode. . The image processing apparatus according to claim 1.
The image processing apparatus, wherein the signal processing unit classifies and outputs the RAW data according to a type of the signal component in the RAW compression mode. The image processing apparatus according to claim 1.
The signal processing unit classifies the RAW data according to the type of the signal component in the RAW compression mode, and outputs it as a G1 component, a G2 component, an R component, and a B component,
The image processing apparatus, wherein the image compression unit compresses an image for each of the G1, G2, R, and B components in the RAW compression mode. The image processing apparatus according to any one of claims 1 to 3,
The signal processing unit
An image processing apparatus that outputs an image signal subjected to color coordinate conversion in the normal compression mode and bypasses the color coordinate conversion in the RAW compression mode. The image processing apparatus according to any one of claims 1 to 4,
The image compression unit
An image processing apparatus, wherein the compression distortion of image compression is lower in the RAW compression mode than in the normal compression mode. The image processing apparatus according to any one of claims 1 to 4,
The image compression unit
In the normal compression mode, the compressed file size is adjusted within the target range,
In the RAW compression mode, the compression distortion is adjusted within a target range. An imaging unit that captures a subject image and generates RAW data in which a plurality of types of signal components constituting a color image are mixedly arranged on a pixel array;
An image processing apparatus according to any one of claims 1 to 6 ,
An RAW data generated by the imaging unit is processed by the image processing apparatus. An image processing program for causing a computer to function as the mode setting unit, the signal processing unit, and the image compression unit according to any one of claims 1 to 6.

Images