JP5858099B2 - Image processing apparatus, image processing method, and imaging apparatus - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an imaging apparatus that are suitable for use in, for example, efficiently compressing and encoding RAW data obtained from imaging elements having various types of arrangement methods.

  2. Description of the Related Art Conventionally, an imaging apparatus using a Bayer array imaging element is generally known. Such an image sensor takes in image light of a subject through a color filter and outputs an image signal according to the intensity of the image light. Then, the subsequent processing unit performs a predetermined process on the image signal, so that the imaging device can display an image on the viewfinder or an external display device.

  Patent Document 1 describes that RAW data (image data before color interpolation) obtained from a Bayer array image sensor is directly compressed by JPEG or the like.

  Patent Document 2 describes a technique for compressing RAW data obtained from Bayer-array image sensors and a specific example using wavelet compression for a G1, G2, R, and B4 component. .

JP 2002-247376 A JP 2003-125209 A

Conventionally, green pixels are compressed after being divided into two components, G1 and G2, so that the pixel positions are alternately displaced in the horizontal direction and the vertical direction, so that the pixel positions are not displaced. This compression is performed after sub-sampling between pixels that have inherently strong correlation as one image to separate screens. For this reason, the correlation between the separated images cannot be used, and the compression efficiency is lowered.
In particular, wavelet transform can achieve very high compression efficiency by dividing the entire screen into subbands, but the conventional method separates it into separate screens, so it did not demonstrate the high compression efficiency inherent in wavelet transform. .

  In addition, by repeating subband division using wavelet transform, images with different resolutions can be obtained from one compression code. For example, when a half resolution image is displayed on a viewfinder or the like with respect to a certain resolution image by the technique described in Patent Document 2, the green is only one of the two divided images. Use. In this case, the merits of using the wavelet transform could not be fully enjoyed, such as simple “pixel thinning” and the image displayed on the viewfinder being affected by aliasing noise.

The conventional technique relates to a method for compressing RAW data obtained from Bayer array image sensors, and is obtained from a double-density Bayer array image sensor or a three-plate image sensor system in which pixels are arrayed obliquely. It does not relate to a technique for compressing RAW data.
Therefore, a different compression method must be used for each pixel array of the image sensor, and hardware cannot be shared.

  Patent Document 1 discloses a technique for performing compression recording without color separation into RGB full pixels. Although this technique has an advantage that the data rate can be suppressed by directly compressing the data without increasing the data rate by color separation, the details of how to compress the data are unknown. If the image is literally compressed as a single image in a Bayer array, the level of each RGB pixel is different in many natural images. Since adjacent pixels are separately painted in RGB, a very large high-frequency component is generated, and compression efficiency does not increase. In other words, the compression noise is expected to become very large.

  An object of the present invention is to efficiently compress and encode RAW data obtained from an image sensor having a pixel array in which pixel positions are alternately shifted in the horizontal direction or the vertical direction for at least one of the three primary colors. To do.

According to the present invention, among image data output from an image sensor having a pixel array in which the pixel positions are alternately shifted in the horizontal direction or the vertical direction for at least one of the three primary colors, the pixel positions are alternately shifted. Sub-band division is performed on color image data. In this subband division, when pixels on two adjacent upper and lower lines are scanned in a zigzag manner, only the subband conversion in the horizontal direction is performed and the subband conversion is not performed on the remaining color image data , or the adjacent left and right two when scanned in the column of the pixel zigzag is not conducted subband transform for the remaining colors image data of the command only subband transform in the vertical direction.

  According to the present invention, even if the pixel positions are shifted alternately in the horizontal or vertical direction, subband division is performed in units of adjacent upper and lower two lines of pixels or adjacent left and right two columns of pixels. For this reason, compression encoding can be performed efficiently by performing compression encoding with the pixel position shifted.

  According to the present invention, image data (RAW data) obtained from an image sensor having a pixel array in which pixel positions are alternately shifted in the horizontal direction or the vertical direction for at least one of the three primary colors is efficiently compressed and encoded. Can be

It is a figure which shows the imaging device which concerns on one embodiment of this invention. It is a figure which shows the example of the pixel arrangement | sequence of an image pick-up element. It is a block diagram at the time of compression encoding of the compression / decompression processing unit. It is a figure which illustrates the division | segmentation level of wavelet transformation. It is a figure which illustrates the division | segmentation level of wavelet transformation. It is a block diagram at the time of compression decoding of a compression / decompression processing unit. It is a figure which shows a prior art. It is the figure which applied the prior art to double density Bayer arrangement. It is a figure which shows the process at the time of the wavelet transformation of a compression / decompression I / F part. It is a figure which shows the process at the time of the reverse conversion of a compression expansion I / F part. It is a figure which shows the wavelet transformation process of the compression expansion I / F part with respect to a double-density Bayer arrangement | sequence. It is a figure which shows another example of the wavelet transformation process of the compression expansion I / F part with respect to a double-density Bayer arrangement | sequence. It is a figure which shows the pixel gravity center position of the subband image after wavelet transformation. It is a figure which shows the wavelet transformation process of the compression expansion I / F part with respect to diagonal arrangement | sequence 3 board system. It is a figure which shows the wavelet transformation process of the compression expansion I / F part with respect to a Bayer arrangement | sequence. It is a figure which shows the output of the compression expansion I / F part with respect to various pixel arrangement | sequences. It is a figure which shows the specific process at the time of the wavelet transformation of the compression / decompression I / F part with respect to a RGB full pixel system. It is a figure which shows the specific process at the time of the reverse conversion of the compression expansion I / F part with respect to a RGB full pixel system. It is a figure which shows the specific process at the time of the wavelet transformation of the compression expansion I / F part with respect to a double-density Bayer arrangement | sequence. It is a figure which shows the specific process at the time of reverse conversion of the compression expansion I / F part with respect to a double density Bayer arrangement | sequence. It is a figure which shows the specific process at the time of the wavelet transformation of the compression expansion I / F part with respect to a diagonal array 3 board system. It is a figure which shows the specific process at the time of reverse conversion of the compression expansion I / F part with respect to a diagonal arrangement | sequence 3 board system. It is a figure which shows the specific process at the time of the wavelet transformation of the compression expansion I / F part with respect to a Bayer arrangement | sequence. It is a figure which shows the specific process at the time of the reverse conversion of the compression expansion I / F part with respect to a Bayer arrangement | sequence. It is a figure which shows the example which a compression / decompression I / F part performs Haar conversion with respect to a double-density Bayer array. It is a figure which shows the example which a compression expansion | extension I / F part performs Haar conversion with respect to a diagonal array 3 board system. It is a figure which shows the example which a compression / decompression I / F part performs Haar conversion with respect to a Bayer arrangement | sequence.

Hereinafter, the best mode for carrying out the invention (hereinafter referred to as an embodiment) will be described. The description will be given in the following order.
1. First embodiment (control of compression encoding or decoding: an example using wavelet transform)
2. Second Embodiment (Control of compression encoding or decoding: example using Haar transform)
3. Modified example

<1. First Embodiment>
[Example of compressing and decoding an image using wavelet transform]

Hereinafter, a first embodiment of the present invention will be described with reference to FIGS.
In the following embodiments, any one obtained from an imaging system using three Bayer array image sensors, double-density Bayer array image sensors, or three image sensors arrayed in an oblique direction, which do not have RGB full pixels. An example applied to the imaging apparatus 10 that efficiently compresses and encodes each R / G / B 3 component also in the RAW data will be described. Hereinafter, the three component signals including RGB pixel elements are abbreviated as “R / G / B 3 components”.

  The imaging apparatus 10 of this example does not have a real pixel, so even if the pixel positions are alternately shifted in the horizontal or vertical direction, compression encoding is performed without reducing the compression efficiency by performing compression encoding without shifting. Can be realized.

Here, the thinned images whose pixel positions are alternately shifted in the horizontal or vertical direction are averaged by combining with the wavelet transform. For this reason, the low frequency sub-band image having half the resolution is an image having RGB full pixels.
That is, a simple so-called color separation (De-Bayer) process is performed by wavelet transform, and the image can be displayed on a monitor (not shown) included in the imaging apparatus 10. This is because when wavelet transform is used, images with different resolutions can be obtained from the same image data. Also, RAW data obtained from an image sensor that requires color separation such as Bayer can be easily displayed on a monitor or the like by performing wavelet transform.

  When an image obtained from an RGB full-pixel image sensor is subjected to wavelet transform, each of RGB is sub-band divided into four images, and the total RGB is decomposed into 12 sub-band images. If the compression coding method of this example is used, RAW data can be handled as a difference in the number of subband images after wavelet transform. For example, the RAW data obtained from the Bayer array is four subband images, the RAW data obtained from the double-density Bayer array is eight subband images, and the RAW data obtained from a three-plate image sensor system in which pixels are arranged diagonally. Then, it can be handled as six subband images. For this reason, the compression encoding process after the wavelet transform can be realized as a common process.

  Further, since a high-resolution image such as a 4K image has a huge amount of data to be handled, some parallel processing is required when performing real-time processing. However, by using the compression encoding method of this example, it can be handled as a difference in the number of subband images as described above. For this reason, it is possible to operate a necessary number of processing blocks in parallel as processing after the image is divided into subbands, and it is possible to suppress the operation speed by the degree of parallelism. Further, as will be described later, the same effect can be obtained not only by the wavelet transform but also by the Haar transform with simple hardware. Note that 4K is an example of high resolution specifications such as 4096 samples × 2160 lines. 2K is an example of a specification with a resolution lower than 4K, such as 2048 samples × 1080 lines.

FIG. 1 shows an example of an imaging apparatus 10 that handles 4K images and 2K images.
The imaging device 10 is an example of an imaging device that can perform image compression / decoding according to the present invention.
The lens block 101 controls the aperture and zoom, and forms an optical image on the image sensor unit 102.

The image sensor unit 102 converts the optical video input from the lens block 101 into a video digital signal and outputs it as recorded RAW data (D102). In this example, any image sensor having a pixel array as shown in FIG. 2 can be used as the image sensor.
<RGB full pixel method>
-RGB 3-plate system that splits into RGB using an optical prism-Single-plate system with a structure that has wavelength sensitivity of light in the depth direction of the sensor-Single-plate system in which one pixel is divided into RGB 3 stripes like a liquid crystal TV <Double density Bayer array>
By doubling the pixel density and arranging the normal Bayer array at an angle of 45 °, G is a full pixel, and the R and B pixels are arranged in an oblique direction (a thick frame in the figure). Is a pixel in a normal Bayer array.
<Slant array 3 plate system>
As one image sensor, pixels are arranged at an angle of 45 degrees, and pixel interpolation is performed between two adjacent horizontal or vertical 2 pixels or 4 pixels, and a thick frame (normal Bayer array in the figure) Is a pixel array premised on interpolating pixels indicated by dotted circles.
A system that uses three of these image sensors to image in combination with an optical prism. Or, a single plate system <Bayer array> with a structure that has the wavelength sensitivity of light in the depth direction of the sensor in the above pixel array.
・ So-called normal Bayer array

The camera signal processing unit 103 also records RAW data (D102) output from the image sensor unit 102 such as a Bayer array, or playback RAW data (D105) read from the recording media unit 108 and subjected to expansion processing described later. Is subjected to predetermined processing. Specifically, the camera signal processing unit 103 creates an RGB full 4K image (so-called color separation) so that it can be confirmed as an image, adjusts the camera image such as white balance and brightness, and records it on the monitor-out unit 104. -Output a playback 4K image (D103).
The monitor-out unit 104 outputs a 4K video signal to an external 4K monitor or the like.

  The compression / decompression I / F unit 105 decomposes recorded RAW data (D102) from an image sensor such as a Bayer array into 2K-band subband images by wavelet transform. The compression / decompression I / F unit 105 has a pixel position of image data output from an image sensor having a pixel array in which pixel positions are alternately shifted in the horizontal direction or the vertical direction for at least one of the three primary colors. It functions as a subband splitting unit that performs subband splitting on adjacent upper and lower two lines of pixels or two adjacent left and right pixels for image data of colors that are alternately shifted.

  Then, the compression / decompression I / F unit 105 performs horizontal wavelet transform on the image data of colors whose pixel positions are alternately shifted by scanning the pixels in units of two adjacent upper and lower lines. Alternatively, the compression / decompression I / F unit 105 performs vertical wavelet transform by scanning pixels in units of two adjacent left and right columns.

In this example, the compression / decompression I / F unit 105 outputs the following subband images.
A 2K band sub-band image that is a low-frequency component in both the horizontal and vertical directions is output as D105-LL.
A 2K band subband image having a high frequency component in the horizontal direction and a low frequency component in the vertical direction is output as D105-HL.
A subband image in the 2K band having a low frequency component in the horizontal direction and a high frequency component in the vertical direction is output as D105-LH.
A 2K band sub-band image that is a high-frequency component in both the horizontal and vertical directions is output as D105-HH.

Depending on the pixel arrangement of the image sensor unit 102, the subband images HL / LH / HH in the 2K band may not be output for each of RGB, and there may be subband images that are not output. Details will be described separately.
Further, the 2K band subband image (D105-LL / HL / LH / HH) read out from the recording media unit 108 and decompressed is subjected to inverse wavelet transform and output as playback RAW data (D105). . Details of this will also be described separately.

  The compression / decompression processing unit 106 compresses each 2K band sub-band image (D105-LL / HL / LH / HH) using a compression encoding method, and each corresponding code stream (D106-LL / HL / (LH / HH). Since the compression / decompression I / F unit 105 uses wavelet transform, the compression / decompression I / F unit 105 is optimal to use JPEG2000 or the like using the same wavelet transform, but uses other existing image compression methods. It doesn't matter.

  The compression / decompression processing unit 106 decompresses each compressed subband image data recorded in the recording media unit 108, and outputs the decompressed subband image (D105-LL / HL / LH / HH) in 2K band. At this time, the compression / decompression processing unit 106 performs a process of compressing and encoding the image data output from the compression / decompression I / F unit 105 for each band and three primary colors divided by the compression / decompression I / F unit 105. It functions as a compression encoding unit that performs in parallel.

The recording medium interface unit 107 performs an interface for accessing the recording medium at high speed and reading / writing compressed image data.
The recording medium unit 108 is a recording medium (an example of a recording unit) for recording and reproducing image compression data, and a nonvolatile memory such as a flash memory is applied.
The viewfinder signal processing unit 109 is an example of a display signal output unit that outputs an input image as a display signal for displaying on the display system. Of the 2K subband images, D105-LL is a low-frequency subband image in both the horizontal and vertical directions, and can be monitored as a 2K size RGB full image. As a result, camera image adjustment such as white balance and brightness can be performed at 2K as with the camera signal processing unit 103. Further, setting information for photographing, peaking processing for facilitating focusing, and the like are performed, and a recording / reproducing 2K image (D109) is output to the viewfinder unit 110.

The viewfinder unit 110 displays the recorded / reproduced 2K image (D109) from the viewfinder signal processing unit 109.
The system control unit 111 has a control software program and controls the entire imaging apparatus 10 according to the program. Further, in response to an input from the operation unit 112, each block is connected to a data bus, exchanges data, and controls settings and states for photographing.
The operation unit 112 receives an operation on the imaging device 10 and transmits the operation to the system control unit 111 as an electrical signal.

  As described above, since the compression / decompression I / F unit 105 uses the wavelet transform, the description will be given first of the compression / decompression processing unit 106 that similarly performs the compression / decompression using the wavelet transform.

FIG. 3 shows a detailed block diagram of the compression / decompression processing unit 106 during compression encoding.
The wavelet transform unit 32 performs wavelet transform on the 2K band sub-band image D105-LL and outputs a wavelet transform coefficient D32-LL.

  The wavelet transform unit 32 is usually realized by a filter bank including a low-pass filter and a high-pass filter. Since a digital filter usually has an impulse response (filter coefficient) having a length of a plurality of taps, it is necessary to buffer in advance an input image or coefficient that can be filtered. Similarly, when the wavelet transform is performed in multiple stages, it is necessary to buffer only the coefficients that allow the wavelet transform coefficients generated in the previous stage to be filtered.

Here, the subband image generated by the wavelet transform will be described.
FIG. 4 shows an example of a subband image. In this wavelet transformation, as shown in FIG. 4, the low frequency components are usually repeatedly transformed and divided, because most of the image energy is concentrated in the low frequency components. This can also be seen from the fact that subband images are formed as the division level is advanced, such as division level = 1 shown in FIG. 5A to division level = 3 shown in FIG. 5B.

  The division level of the wavelet transform in FIG. 4 is 3, and as a result, a total of 10 subband images are formed. Here, in FIG. 4, L and H represent the low frequency and the high frequency, respectively, and the numbers before L and H represent the division level. That is, for example, 1LH represents a subband image of division level = 1 in which the horizontal direction is a low frequency and the vertical direction is a high frequency.

Returning to the description of FIG. The wavelet transform coefficient D32-LL is then quantized by the quantizing unit 33 and the quantized coefficient D33-LL is output. As the quantization means here, the scalar quantization used in JPEG2000 may be used. As shown in the following (Equation 1), the value obtained by dividing the wavelet transform coefficient W by the quantization step size Δ may be the value of the quantization coefficient q.
q = W / Δ (Equation 1)

The quantization step size ΔD37-LL is given from a code amount measuring unit 37 described later.
The quantization coefficient D33-LL is then output to the entropy coding unit 35, and the entropy coding unit 35 performs an operation of compressing the quantization coefficient D33-LL using an arbitrary information source compression unit. As the entropy encoding means, commonly used Huffman encoding (used in MPEG and JPEG, refer to the Huffman encoding table created in advance according to the appearance frequency of the symbols appearing in the data, Code generation method) and arithmetic coding (methods adopted in H.264 and JPEG2000) may be used. At this time, although details are not described here, the quantization coefficient may be combined with EBCOT (Embedded Block Coding with Optimal Truncation) which is entropy coding in bit plane units, as in JPEG2000.

  The result encoded by the entropy encoding unit 35 is output as an encoded code stream D35-LL, becomes an output D106-LL of the compression / decompression processing unit 106, and is also input to the code amount measurement unit 37.

The code amount measuring unit 37 compares the code amount within one frame of the encoded code stream D35-LL with the target code amount D36-LL given from the control unit 36. When the target code amount is likely to be exceeded, the quantization step size D37-LL of the quantization unit 33 is changed to be increased by one step.
On the contrary, when the accumulation of the code amount within one frame of the encoded code stream D35-LL is likely to be less than the target code amount, the quantization step size D37-LL of the quantization unit 33 is decreased by one step. Change as follows.

  The above is the operation at the time of compression encoding of the compression / decompression processing unit 106. In addition to the subband image LL component of 2K band, the compression / decompression processing unit 106 receives HL component, LH component, and HH component. The

  As for the HL / LH / HH component, similarly to the LL component, the wavelet transform unit 32 may be used to increase the division level. However, in the example of FIG. 3, the first level wavelet transform has already been performed in the compression / decompression I / F unit 105. For this reason, only the low-frequency component subband image D105-LL is iteratively transformed and divided in accordance with the normal wavelet transform using the property that much of the image energy shown in FIG. 4 is concentrated in the low-frequency component. It is set as the mode to be done.

Note that, for example, processing from wavelet transform to entropy encoding may be realized by a single circuit by hardware. In this case, there is no restriction on the configuration of the compression / decompression processing unit 106 using four existing parallel circuits without developing a circuit that does not perform wavelet transformation.
In the example of FIG. 3, only the quantization and entropy coding are performed on the subband images D105-HL, D105-LH, and D105-HH in the 2K band other than the LL component, and the wavelet division level is not increased. .

  The subband images D105-HL, D105-LH, and D105-HH in the 2K band other than the LL component are output as encoded code streams D35-HL, D35-LH, and D35-HH, respectively. Then, the outputs D106-HL, D106-LH, and D106-HH of the compression / decompression processing unit 106 are obtained. In addition, the code amount measuring unit 37 accumulates the code amount in one frame of each encoded code stream, while the target code amounts D36-HL, D36-LH, D36-HH given from the control unit 36 are Make a comparison. Similarly to the LL component, when the code amount in one frame of the encoded code stream is likely to exceed the target code amount, the quantization step size of the quantization unit 33 is changed to be increased by one step. . On the other hand, when the accumulation of the code amount within one frame of the encoded code stream is likely to be less than the target code amount, the quantization step size of the quantization unit 33 is changed to be one step smaller.

As described above, the code amount for each sub-band image in the 2K band is controlled.
It has been described that the control unit 36 sets the target code amount for each subband image in the 2K band in the corresponding code amount measurement unit 37 in advance. However, for example, information on the code amount integration status is sent from each code amount measurement unit 37 to the control unit 36, and the target code amount for each subband is dynamically accommodated according to each code amount integration status. May be changed. Since a natural image is an image containing a lot of low-frequency components, or conversely an image containing many high-frequency components, optimal code amount control according to the properties of the image is possible.
The above is the description of the operation at the time of compression encoding.

Next, FIG. 6 shows a detailed block diagram of the compression / decompression processing unit 106 at the time of compression decoding, and the operation at the time of compression decoding will be described.
The entropy decoding unit 38 to which the encoded code streams D106-LL, D106-HL, D106-LH, and D106-HH are input performs decoding according to the means corresponding to the entropy encoding described in FIG. As a result of the entropy decoding, quantization coefficients D38-LL, D38-HL, D38-LH, and D38-HH are generated.

The quantization coefficients D38-LL, D38-HL, D38-LH, and D38-HH are converted from the quantization coefficients D38-LL, D38-HL, D38-LH, and D38-HH by the inverse quantization unit 39 to the wavelet transform coefficient D39-. LL, D39-HL, D39-LH, and D39-HH. The inverse quantization means here is the reverse operation of (Equation 1) and can be expressed by the following (Equation 2).
W = q × Δ (Equation 2)
(W is a wavelet transform coefficient, q is a quantization coefficient, Δ is a quantization step size)

The wavelet transform coefficient D39-LL is returned to the 2K band subband image D105-LL of the LL component by the wavelet inverse transform unit 40 and output.
When wavelet subdivision is performed on subband images D105-HL, D105-LH, and D105-HH in 2K bands other than the LL component at the time of encoding, the wavelet inverse transform unit 40 is provided for each subband. Obviously, it is only necessary to return to the subband images of each 2K band.
The above is the description of the operation at the time of compression decoding.

  Here, with reference to FIG. 7, the problems of the technique described in the conventional patent document 2 are organized.

FIG. 7A is an example of a 4K Bayer RAW image using a normal Bayer array.
Although there are 4K pixels, in a normal Bayer array, this is separately applied to RGB as shown in the figure, and only the wavelength component of light corresponding to the RGB color filter is converted from light to an electrical signal in each pixel. In a normal Bayer array, G is arranged in a checkered pattern.

FIG. 7B shows an example of components of four color components of R pixel, G1 pixel, G2 pixel, and B pixel obtained by separating the pixels of the 4K Bayer RAW image.
In the technique described in Patent Document 2, the 4K Bayer RAW image described above is regarded as a component of four color components, that is, an R pixel, a G1 pixel, a G2 pixel, and a B pixel.

FIG. 7C shows an example in which the components of the four color components shown in FIG. 7B are collected for each pixel.
In the technique described in Patent Document 2, the pixels included in the components of the four color components shown in FIG. 7B are collected into a 2K R image, a 2K G1 image, a 2K G2 image, and a 2K B image, It is recorded after compression encoding using wavelet transform.

  Here, it is known that the G1 pixel and the G2 pixel are originally adjacent in the oblique direction, and the G1 pixel and the G2 pixel have a very high correlation. Nonetheless, the technique described in Patent Document 2 performs compression encoding as if it were two independent images (or two color components), namely the G1 image and the G2 image. Resulting in. In particular, the wavelet transform achieves high compression efficiency in terms of subband transform of the entire screen, but it cannot be said that the combination of the conventional technology and the wavelet transform enjoys the original high compression rate of the wavelet transform. I want.

  Further, as shown in FIG. 7B, since “thinning” is performed by sub-sampling of the G1 image and the G2 image, when only the G1 image or the G2 image is displayed on the 2K monitor, the aliasing distortion is caused by the sampling theorem. It is obvious that will occur. Thus, it is conceivable to use both the G1 image and the G2 image to generate a 2K average image by signal processing and display it on a monitor. However, data twice as large as 2K must be accessed. However, since the wavelet transform is subband division, it has the ability to display on a monitor if only the 2K low frequency component is accessed. In this respect as well, it is difficult to say that the technique described in Patent Document 2 fully enjoys the merits of wavelet transform.

FIG. 8 shows an example in which the prior art is applied to a double-density Bayer array.
FIG. 8A shows a normal Bayer array in which the pixel density is doubled and the pixels are arranged at an angle of 45 degrees.
The feature of this arrangement is that the G image has 4K full pixels, and the R image and the B image are slanted, but 4K resolution is obtained in the horizontal and vertical directions. Since the pixel density is twice that of a normal Bayer, it will be referred to as a “double density Bayer array” in the following description.

  Here, consider a case where a 4K RAW image using this double-density Bayer array is compressed according to the technique described in Patent Document 2. In this case, unlike the Bayer array, as shown in FIG. 8B, G has all 4K pixels, but the R image and B image have a checkered pattern. For this reason, as shown in FIG. 8C, the R1 pixel, the R2 pixel, the G pixel (however, 4K resolution), the B1 pixel, and the B2 pixel are considered as components of five color components. Then, as shown in FIG. 8D, the pixels are collected into a 2K R1 image, a 2K R2 image, a 4K G image, a 2K B1 image, and a 2K B2 image, and a compression code using wavelet transform for each image. Will be recorded.

  In the Bayer array, only the G image is sub-sampled and compressed into two screens, but in the double-density Bayer array, both the R image and the B image are sub-sampled and separated into two screens. Further, in the Bayer array, the G image with a high pixel density is separated into two screens. Therefore, even if the compression efficiency of the G image is reduced, the compressed image of the R image and the B image with the originally high pixel density and the low pixel density. Balance with quality.

On the other hand, in the double-density Bayer array, both the R image and the B image having a low pixel density are further sub-sampled into two screens, so that the pixel density is low and the compression efficiency is lowered. The balance further deteriorates with respect to the image quality.
Further, as described in the Bayer array G image, there is a concern that aliasing may occur when the subsampled R image and B image are displayed as they are on a 2K monitor.

  On the other hand, the imaging apparatus 10 of the present example can be expected to make a significant improvement utilizing the original characteristics of the wavelet transform, particularly regarding the RAW data compression of the R image and B image in the double density Bayer array. Furthermore, the same technique can be used from a normal Bayer array to RGB full pixels.

Next, a processing example at the time of wavelet transform of the compression / decompression I / F unit 105 will be described with reference to FIG.
The compression / decompression I / F unit 105 can exert its effect to the maximum when the wavelet transform is used. Therefore, although the basic configuration is the same as that of the wavelet transform unit 32 of the compression / decompression processing unit 106, the processing of the compression / decompression I / F unit 105 will be described again with reference to FIG.

The recording RAW data D102 is input as an input image of the compression / decompression I / F unit 105. Here, assuming that the 4K RGB full pixels shown in FIG. 2 are input, the same processing is performed on the R / G / B 3 component of 4096 × 2160 size.
First, horizontal wavelet transform is performed on the input image. The wavelet transform is composed of a low-pass filter (LPF) and a high-pass filter (HPF), and 2: 1 downsampling (represented by a downward arrow and numeral 2 in the figure) is performed. As a result, a 2048 × 2160 low-frequency subband image (L) and a high-frequency subband image (H) having a half size in the horizontal direction are obtained.

  Next, vertical wavelet transform is performed on the 2048 × 2160 low-frequency subband image (L). In the wavelet transform, 2: 1 down-sampling is performed through a low-pass filter (LPF) and a high-pass filter (HPF), respectively. By this processing, a 2048 × 1080 low-frequency subband image (L, L) D105-LL and a high-frequency subband image (L, H) D105-LH having a half size in the vertical direction are obtained.

  In addition, vertical wavelet transform is performed on the 2048 × 2160 high-frequency subband image (H). In the wavelet transform, 2: 1 down-sampling is performed through a low-pass filter (LPF) and a high-pass filter (HPF), respectively. By this processing, a 2048 × 1080 low-frequency subband image (H, L) D105-HL and a high-frequency subband image (H, H) D105-HH having a half size in the vertical direction are obtained.

As described above, the compression / decompression I / F unit 105 performs wavelet transform on the R / G / B 3 component of 4096 × 2160 size to obtain the following three components.
・ Subband image (L, L) D105-LL of 2048 × 1080 size that passed through low-pass filter in both horizontal and vertical directions is R / G / B each of three components ・ Low-pass filter in horizontal direction, vertical direction Subband image (L, H) D105-LH of 2048 × 1080 size that has passed through a high-pass filter is R / G / B each of three components. 2048 that has passed through a high-pass filter in the horizontal direction and a low-pass filter in the vertical direction × 1080 size subband image (H, L) D105-HL is a 2048 × 1080 size subband image (H, H) through which a high-pass filter is passed in each of the three components R / G / B in the horizontal and vertical directions. ) D105-HH is R / G / B each 3 components In this way, 4 types of 2048 × 1080 size subbands The image is converted into subband images of 2048 × 1080 size of R / G / B each 3 components worth of total 12.

Next, the processing at the time of wavelet inverse transformation of the compression / decompression I / F unit 105 will be described with reference to FIG.
As input images of the compression / decompression I / F unit 105, sub-band images (D105-LL / HL / LH / HH) of 2048 × 1080 size read out from the recording media unit 108 and decompressed are respectively R / G / B Three components are input.

  2048 × 1080 size subband image (L, L) D105-LL that has passed a low-pass filter in both the horizontal and vertical directions, and a 2048 × 1080 size that has a low-pass filter in the horizontal direction and a high-pass filter in the vertical direction Wavelet inverse transform in the vertical direction is performed on the subband image (L, H) D105-LH. The inverse wavelet transform performs 1: 2 upsampling on D105-LL (indicated by the up arrow and number 2 in the figure), and up to 1: 2 on D105-LH in the low-pass filter (LPF). Sampling and passing through a high pass filter (HPF). Then, by combining the two, a 2048 × 2160 low-frequency subband image (L) having a double size in the vertical direction is obtained.

  Further, a 2048 × 1080 size subband image (H, L) D105-HL that has passed through a high-pass filter in the horizontal direction and a low-pass filter in the vertical direction, and 2048 × that has passed through a high-pass filter in both the horizontal and vertical directions. Wavelet inverse transform in the vertical direction is performed on the 1080-size subband image (H, H) D105-HH. In the wavelet inverse transform, D105-HL performs 1: 2 upsampling and passes through a low-pass filter (LPF), and D105-HH performs 1: 2 upsampling and passes through a high-pass filter (HPF). Then, by combining them, a 2048 × 2160 high-frequency subband image (H) having a vertical size twice as large is obtained.

  Next, the wavelet inverse transform in the horizontal direction is performed on the 2048 × 2160 low-frequency subband image (L) and the 2048 × 2160 high-frequency subband image (H) on which the vertical wavelet inverse transform has been performed. . In the wavelet inverse transform, the low-frequency subband image (L) is up-sampled 1: 2 to the low-pass filter (LPF), and the high-frequency sub-band image (H) is up-sampled 1: 2 to the high-pass filter (LP). HPF). Then, by combining them, 4096 × 2160 playback RAW data D105 having a size twice as large in the horizontal direction is obtained.

  As described above, the processing at the time of inverse wavelet conversion of the compression / decompression I / F unit 105 is read from the recording media unit 108 and decompressed 2048 × 1080 size subband image (D105-LL / HL / LH / HH). In this process, the subband images are respectively decoded for R / G / B 3 components and subjected to inverse wavelet transform in the vertical and horizontal directions to be decoded into R / G / B 3 components of 4096 × 2160 size. Yes.

  Now, how the compression / expansion I / F unit 105 of this example processes the double-density Bayer array shown in FIG. 2 will be described with reference to FIG.

FIG. 11A is a double-density Bayer in which a normal Bayer array is doubled and arranged at an angle of 45 degrees as described above, and the G image is a 4K full pixel, the R image and the B image. Although it is empty at an angle, 4K resolution is obtained in the horizontal and vertical directions.
At this time, the imaging element unit 102 is a double-density Bayer array that is a pixel array that doubles the pixel density of the Bayer array and that is diagonally arranged at 45 °.

  Then, the compression / decompression I / F unit 105 unites two adjacent upper and lower lines for R and B image data among RGB image data output from the image sensor unit 102 having a double-density Bayer array. The pixel is scanned and horizontal wavelet transform is performed, or the pixel is scanned in units of two adjacent left and right columns and the vertical wavelet transform is performed, and the image is decomposed into R and B subband images. The G image data is decomposed into G subband images by performing horizontal wavelet transform in units of pixels of one line and vertical wavelet transform in units of pixels of one column. .

  Specifically, first, G will be described. Since the G image has 4K full pixels, it is clear that the same processing as that described in FIG.

  That is, the compression / decompression I / F unit 105 performs horizontal wavelet transform on a 4096 × 2160 size G image as shown in FIG. 11B. Then, as shown in FIG. 11C, a 2048 × 2160 low-frequency sub-band G image (L) and a high-frequency sub-band G image (H) that are halved in the horizontal direction are obtained. A portion surrounded by a dotted line in the figure is an example of a 5 × 3 reversible wavelet filter (5 taps for a low-pass filter and 3 taps for a high-pass filter) defined by JPEG2000. A pixel range in which a 5-tap filter (a low-pass filter) centering on the marked pixel) is shown. By performing this every two pixels in the horizontal direction, 2: 1 downsampling is realized. For details, refer to the JPEG2000 standard.

Next, as shown in FIG. 11C, a vertical wavelet transform is performed on each of the 2048 × 2160 low-frequency subband G image (L) and high-frequency subband G image (H), which are half the size in the horizontal direction. Apply. Then, as shown in FIG. 11D, the following subband images are obtained.
A 2048 × 1080 size subband G image (LL) that has passed through a low-pass filter in both the horizontal and vertical directions.
A 2048 × 1080 size subband G image (LH) through a low-pass filter in the horizontal direction and a high-pass filter in the vertical direction.
A 2048 × 1080 size subband G image (HL) that has passed through a high-pass filter in the horizontal direction and a low-pass filter in the vertical direction.
A 2048 × 1080 size subband G image (HH) that has passed through a high-pass filter in both the horizontal and vertical directions.
In this way, the G images constituting the 4K double density Bayer RAW image are converted into four types of 2048 × 1080 size subband G images.

Here, the R image and the B image are examined. Since the R image and the B image have a checkered pattern, if the method described with reference to FIG. 8 is used, the pixel density is low and the compression efficiency is lowered as described above. However, the balance will be further lost.
Therefore, the compression / decompression I / F unit 105 of this example performs wavelet transform on the R image and the B image in units of two adjacent upper and lower lines.
Specifically, as shown in FIG. 11B, the compression / decompression I / F unit 105 scans pixels in the W shape in units of two adjacent upper and lower lines, as if it were pixel data of one line. Perform wavelet transform.
In addition, the compression / decompression I / F unit 105 scans the pixels in the M-type in units of two adjacent upper and lower lines, and performs wavelet conversion as if it were pixel data of one line.

By setting the upper and lower two lines as a unit, since one line can be regarded as having 4096 actual pixels, the R image and the B image can be handled as 4096 × 1080 images.
This 4096 × 1080 R image is wavelet transformed in the horizontal direction,
2048 × 1080 size subband R image that has passed through the low-pass filter,
2048 × 1080 size subband R image through high-pass filter,
Convert to
Similarly, for the B image, this 4096 × 1080 B image is wavelet transformed in the horizontal direction,
2048 × 1080 size subband B image that has passed through the low-pass filter,
2048 × 1080 size subband B image that has passed through the high-pass filter,
Convert to

Further, when the wavelet transform in the horizontal direction is performed, the subband image already has a size of 2048 × 1080, and the size matches that of the G subband image. Therefore, no wavelet transform is performed in the vertical direction.
This means that the subband images of 2048 × 1080 size are aligned in RGB, and a full RGB image of 2048 × 1080 has been obtained, so that simple so-called color separation has been performed. become.
As a result, the subband image that is output from the compression / decompression I / F unit 105 has the same size regardless of whether it is a 4K full pixel system or a 4K double-density Bayer array. It can be a process. This will be described in detail separately.

In FIG. 11D, the R image and the B image are described as follows for the purpose of explaining that only the horizontal wavelet transform is performed, in distinction from another method described later.
・ Subband R image (LL) of 2048 × 1080 size through low-pass filter
・ Subband R image (HL) of 2048 × 1080 size through high-pass filter
・ Subband B image of 2048 × 1080 size that passed through low-pass filter, Subband B image (LL)
・ Subband B image (HL) of 2048 × 1080 size through high-pass filter
The compression / expansion I / F unit 105 has been described as scanning the pixel in the R type for the W pixel and the B pixel for the M type. However, if the starting pixel positions of the R pixel and the B pixel are different, the W type and the M type are scanned. Is clearly the opposite.

  As described above, FIG. 11 illustrates the method for solving the problem by performing wavelet transform on the R image and the B image in the double-density Bayer array in units of two adjacent upper and lower lines. However, as another method, a method of solving the problem by performing wavelet transform on the R image and B image in the double-density Bayer array in units of two adjacent pixels will be described with reference to FIG.

  First, regarding the G image, since the G image has 4K full pixels, the process is the same as the process described in FIG.

  The R image and B image of the double-density Bayer array are also apparent from the figure, but will be described in order.

At this time, the compression / decompression I / F unit 105 performs two adjacent left and right columns for R and B image data out of RGB image data output from the image sensor unit 102 having a double-density Bayer array. A pixel is scanned as a unit to perform wavelet transform in the vertical direction. The G image data is subjected to horizontal wavelet transform in units of one line of pixels and vertical wavelet transform in units of one column of pixels.
Specifically, as shown in FIG. 12B, the R pixel and the B pixel are not subjected to wavelet transformation in the horizontal direction, but are sent to the wavelet transformation in the vertical direction as they are. At this time, as shown in FIG. 12C, the compression / decompression I / F unit 105 scans the pixels in the leftward W-type in units of two adjacent pixels on the left and right sides, and is pixel data arranged in one column. Perform wavelet transform as if.
Further, the compression / decompression I / F unit 105 scans the pixels in the rightward W-type in units of two adjacent left and right pixels with respect to the B pixel, and performs wavelet conversion as if it were pixel data arranged in one column.

By setting the left and right pixels as a unit, it can be considered that the vertical direction has 2160 real pixels, so that the R image and the B image can be handled as 2048 × 2160 images.
This 2048 × 2160 R image is wavelet transformed in the vertical direction. At this time, the compression / decompression I / F unit 105
2048 × 1080 size subband R image that has passed through the low-pass filter,
2048 × 1080 size subband R image through high-pass filter,
Convert to
Similarly for the B image, the 2048 × 2160 B image is wavelet transformed in the vertical direction. At this time, the compression / decompression I / F unit 105
2048 × 1080 size subband B image that has passed through the low-pass filter,
2048 × 1080 size subband B image that has passed through the high-pass filter,
Convert to

When the wavelet transform in the vertical direction is performed, the subband image has a size of 2048 × 1080, and the size matches that of the G subband image.
As a result, the subband image that is output from the compression / decompression I / F unit 105 has the same size regardless of whether it is a 4K full pixel system or a 4K double density Bayer array. Can be processed. This will be described in detail separately.

In FIG. 12D, subband images are described as follows for the purpose of distinguishing from other methods described later, in order to explain that the R image and the B image are only subjected to the wavelet transform in the vertical direction.
・ Subband R image (LL) of 2048 × 1080 size through low-pass filter
・ Subband R image (LH) of 2048 × 1080 size through high-pass filter
・ Subband B image of 2048 × 1080 size that passed through low-pass filter, Subband B image (LL)
-Subband B image (LH) of 2048x1080 size that passed through the high-pass filter

  Here, the compression / decompression I / F unit 105 has been described with reference to FIG. 12 assuming that the R pixel is scanned in the left-facing W shape and the B pixel is scanned in the right-facing W shape, but the origin pixel positions of the R and B pixels are described. Obviously, the left-facing W type and the right-facing W type are reversed.

As described above, how to perform wavelet transform processing on the double-density Bayer array has been described with reference to FIGS. 11 and 12.
Next, the pixel barycentric position of the 2048 × 1080 size subband image that has passed through the low-pass filter after wavelet transform will be described with reference to FIG.
Here, a description will be given by using, as an example, 5 × 3 reversible wavelet filter coefficients (−1/8, 2/8, 6/8, 2/8, −1/8) of JPEG2000.

  FIG. 13A shows an example in which a 5 × 3 reversible wavelet filter is applied to a G image. First, the filter coefficient of (-1/8, 2/8, 6/8, 2/8, -1/8) is applied to the five pixels in the horizontal direction. 6/8 is multiplied by the third pixel, and the symmetry coefficient is multiplied by the left and right pixels around this, so that the center of gravity of the pixel in the horizontal direction matches the position of the third pixel.

Next, a filter coefficient of (−1/8, 2/8, 6/8, 2/8, −1/8) is applied to the five pixels in the vertical direction. 6/8 is multiplied by the third pixel, and the symmetry coefficient is multiplied by the upper and lower two pixels around this, so that the center of gravity of the pixel in the vertical direction also matches the position of the third pixel.
That is, the pixel centroid position of the 2048 × 1080 size subband image obtained by passing the low-frequency wavelet filter through the G image is a pixel in which the filter coefficient is multiplied by 6/8 in both the horizontal and vertical directions (in the drawing). The circle mark ○).

  FIG. 13B shows an example in which a 5 × 3 reversible wavelet filter is applied to an R image. As described with reference to FIG. 11, the compression / decompression I / F unit 105 scans pixels in the W shape in units of two adjacent upper and lower lines, as if the R image is one line of pixel data. Wavelet transform.

The five pixels are multiplied by filter coefficients of (-1/8, 2/8, 6/8, 2/8, -1/8). 6/8 is multiplied by the third pixel, and the symmetry coefficient is multiplied by the left and right pixels around this, so that the center of gravity of the pixel in the horizontal direction matches the position of the third pixel.
For the R image, the wavelet transform is not performed in the vertical direction. However, when attention is paid to the wavelet coefficient in the horizontal direction as described above, for the three pixels on the upper line, (−1/8, 6 / 8, -1 / 8) filter coefficients are applied. For this reason, the weight of the coefficient of the three pixels on the upper line is
-1 / 8 + 6 / 8-1 / 8 = 1/2
It turns out that it is.

Next, since the filter coefficients of (2/8, 2/8) are applied to the two pixels on the lower line, the weight of the coefficients of the two pixels on the lower line is
2/8 + 2/8 = 1/2
It turns out that it is.
From the above, it can be seen that the center of gravity of the pixel in the vertical direction of the R image is exactly the middle position between the two lines in which the pixel is scanned in the W shape in units of two adjacent upper and lower lines.
That is, the pixel centroid position of the 2048 × 1080 size subband image obtained by passing the low-frequency wavelet filter with respect to the R image is the pixel position multiplied by the 6/8 filter coefficient in the horizontal direction, and in the vertical direction. Is the position between the lines in the unit of two adjacent upper and lower lines (circles in the figure).

  FIG. 13C shows an example in which a 5 × 3 reversible wavelet filter is applied to the B image. As described with reference to FIG. 11, the compression / expansion I / F unit 105 scans pixels in the M-type in units of two adjacent upper and lower lines, as if the B image is one line of pixel data. Perform wavelet transform.

The five pixels are multiplied by filter coefficients of (-1/8, 2/8, 6/8, 2/8, -1/8). 6/8 is multiplied by the third pixel, and the symmetry coefficient is multiplied by the left and right pixels around this, so that the center of gravity of the pixel in the horizontal direction matches the position of the third pixel.
For the B image, the wavelet transform is not performed in the vertical direction. However, when attention is paid to the wavelet coefficient in the horizontal direction, (-1/8, 6) for the three pixels on the lower line. / 8, -1/8) filter coefficients are applied. For this reason, the weight of the coefficient of the three pixels on the lower line is
-1 / 8 + 6 / 8-1 / 8 = 1/2
It turns out that it is.

Next, since the filter coefficients of (2/8, 2/8) are applied to the two pixels on the upper line, the coefficient weight of the two pixels on the upper line is
2/8 + 2/8 = 1/2
It turns out that it is.
From the above, it can be seen that the center of gravity of the pixel in the vertical direction of the B image is exactly the middle position between two lines in which the pixel is scanned in an M shape in units of two adjacent upper and lower lines.
That is, the pixel centroid position of the 2048 × 1080 size subband image obtained by passing the B image through the low-frequency wavelet filter is the pixel position multiplied by the 6/8 filter coefficient in the horizontal direction. In the vertical direction, the position is a position between adjacent lines in the upper and lower two lines (circle mark in the figure).

  In summary, the pixel centroid position of the 2048 × 1080 size subband image that has passed through the low-frequency wavelet filter is as shown in FIG. 13D. At this time, it can be seen that the R image and the B image are at the same position, while the G image is at the same position in the vertical direction, and is slightly shifted by 1/2 pixel in the horizontal direction.

Here, one of the features of subband division using a wavelet filter such as JPEG2000 is that an image size of 1 / ²ⁿ resolution can be obtained from one compressed stream. This indicates that even in the double-density Bayer array, when only the low-frequency subband images having half the horizontal and vertical image sizes are displayed, the RGB pixel barycentric positions are almost the same. Yes. That is, it can be displayed as a low-frequency subband image with almost no color shift without matching the pixel centroid position.

Of course, if the G image is corrected in the horizontal direction by a half pixel, RGB completely matches. Also in this case, since only the horizontal direction is corrected by 1/2, there is an effect that the hardware scale is small.
Also, as shown in FIG. 12, even when wavelet transform is performed only in the vertical direction for the R image and the B image, although not shown in detail, considering the same as in FIG. 13, the R image and the B image are It can be seen that the same position, in contrast, the G image is the same position in the horizontal direction and is slightly shifted by 1/2 pixel in the vertical direction.
The above is the details of the process for the double-density Bayer array.

  Next, an example of compression encoding processing for the diagonally arranged three-plate method of FIG. 2 will be described with reference to FIG.

In FIG. 14A, as one image sensor unit 102, pixels are arranged at an angle of 45 degrees, and pixel interpolation is performed between two adjacent horizontal or vertical two pixels or between four pixels. This is a pixel array premised on interpolating pixels indicated by circles.
At this time, the image sensor unit 102 is a diagonal array three-plate system that is a pixel array on the assumption that pixels are arrayed at an angle of 45 ° and interpolation is performed between pixels adjacent in the horizontal direction or the vertical direction.

  Here, a system that uses three image sensor sections 102 having the above pixel arrangement and combines them with an optical prism, or a single plate having a structure that has wavelength sensitivity of light in the depth direction of the sensor with this pixel arrangement A method of applying the present invention to a system imaging system will be described.

First, the pixels indicated by dotted circles in the figure are premised on interpolation from surrounding pixels, and therefore can be considered not to exist as actual pixels. That is, as shown in FIG. 14B, since it is considered that there is no pixel marked in gray, this can be considered the same as the R or B pixel in the double-density Bayer array described with reference to FIG.
The difference is that the R image, the B image, and the G image all have the same pattern.
Therefore, wavelet transform may be performed in units of two adjacent upper and lower lines for the R image, the B image, and the G image.

  At this time, the compression / decompression I / F unit 105 scans the pixels in units of two adjacent upper and lower lines with respect to all of the RGB image data from the image sensor unit 102 of the diagonally arranged three-plate method, and performs horizontal scanning. Wavelet transformation is performed, or vertical wavelet transformation is performed by scanning pixels in units of two adjacent left and right columns.

Specifically, as shown in FIG. 14B, the compression / decompression I / F unit 105 scans pixels in the W-type in units of two adjacent upper and lower lines, and performs wavelet transform as if it were pixel data of one line. I do. Depending on the starting position of the pixel, it is possible to scan the pixel in the M-type in units of two adjacent upper and lower lines, and perform wavelet transform as if it were pixel data of one line.
By setting the upper and lower two lines as a unit, since one line can be regarded as having 4096 actual pixels, it can be handled as a 4096 × 1080 image.
This 4096 × 1080 RGB image is wavelet transformed in the horizontal direction,
2048 × 1080 size subband image that has passed through the low-pass filter,
2048 × 1080 size subband image that has passed through the high-pass filter,
The RGB images are converted respectively.

Further, when the wavelet transform in the horizontal direction is performed, it is already a subband image of 2048 × 1080 size, and the wavelet transform is not performed in the vertical direction.
This means that the subband images of 2048 × 1080 size are aligned in RGB, and a full RGB image of 2048 × 1080 has been obtained, so that simple so-called color separation has been performed. become.
As a result, the subband image that is output from the compression / decompression I / F unit 105 has the same size in both the 4K full pixel method and the 4K diagonally arranged three-plate method. Can be processed. This will be described in detail separately.

In FIG. 14D, a subband image having a size of 2048 × 1080 is described as follows for the purpose of explaining that the image has been subjected only to horizontal wavelet transform, in distinction from another method described later.
・ Subband R image (LL) of 2048 × 1080 size through low-pass filter
・ Subband R image (HL) of 2048 × 1080 size through high-pass filter
・ Subband G image (LL) of 2048 × 1080 size through low-pass filter
・ Subband G image (HL) of 2048 × 1080 size through high-pass filter
・ Subband B image of 2048 × 1080 size that passed through low-pass filter, Subband B image (LL)
・ Subband B image (HL) of 2048 × 1080 size through high-pass filter

  In FIG. 14, the process of performing wavelet transform on an adjacent upper and lower two line unit has been described for an image with a diagonally arranged three-plate method, but as another method, wavelet transform can be performed on an adjacent left and right two pixel unit. However, since the processing is exactly the same as the processing described in the processing of the R image and B image in the double-density Bayer array described in FIG.

In addition, the pixel centroid position of the 2048 × 1080 size subband image that has passed through the low-pass filter after wavelet transform in the double-density Bayer array is described with reference to FIG.
However, in the diagonal array three-plate method, the pixel positions are optically adjusted so that RGB is also applied to the pixel centroid position of the 2048 × 1080 size subband image that has passed through the low-pass filter after wavelet transform. Since it is clear that they are aligned, they are omitted here.
The above is the details of the processing for the diagonally arranged three-plate method.

  Next, a processing example for the normal Bayer array shown in FIG. 2 will be described with reference to FIG.

FIG. 15A shows a so-called normal Bayer array.
At this time, the image sensor unit 102 is a Bayer array.

In FIG. 15B, the positions where the actual pixels do not exist are shown in gray, separated into RGB. The R image and the B image are already decimated images of 2048 × 1080 size in a separated state.
As is apparent from the drawing, the G image can be considered to be the same as the R pixel or B pixel of the double-density Bayer array described with reference to FIG.
Therefore, the G image may be subjected to wavelet transform in units of two adjacent upper and lower lines.

At this time, the compression / decompression I / F unit 105 does not perform wavelet transform on R and B image data out of RGB image data from the Bayer array image sensor unit 102. For the G image data, the pixel is scanned in units of two adjacent upper and lower lines and horizontal wavelet transform is performed, or the pixels are scanned in units of two adjacent left and right columns and the vertical direction is scanned. Perform wavelet transform.
Specifically, as shown in FIG. 15B, the compression / decompression I / F unit 105 scans pixels in the M-type in units of two adjacent upper and lower lines, and performs wavelet transform as if it were pixel data of one line. I do.

  Depending on the starting position of the pixel, it is also possible to scan the pixel in a W-type in units of two adjacent upper and lower lines, and perform wavelet transformation as if it were pixel data of one line.

By setting the upper and lower two lines as a unit, since one line can be regarded as having 4096 actual pixels, it can be handled as a 4096 × 1080 image.
This 4096 × 1080 G image is wavelet transformed in the horizontal direction,
2048 × 1080 size subband G image that has passed through the low-pass filter,
2048 × 1080 size subband G image that has passed through the high-pass filter,
Convert to

Further, when the wavelet transform in the horizontal direction is performed, it is already a subband image of 2048 × 1080 size, and the wavelet transform is not performed in the vertical direction.
This means that the R image and the B image are thinned out, but the subband images of 2048 × 1080 size are arranged in RGB, and the RGB full image of 2048 × 1080 is obtained. Thus, simple so-called color separation is performed.
As a result, the subband image that is output from the compression / decompression I / F unit 105 has the same size in both the 4K full pixel method and the 4K normal Bayer array. It can be. This will be described in detail separately.

In FIG. 15D, the R image and the B image are thinned out. However, in order to make the processing common to other methods, the G image is distinguished from another method described later, and the image is a horizontal wavelet. In order to explain that only conversion was performed, the following is described.
That is, an R image of 2048 × 1080 size is converted into a subband R image (LL),
A 2048 × 1080 size subband G image that has been passed through a low-pass filter, a subband G image (LL),
A 2048 × 1080 size subband G image that has been passed through a high-pass filter, a subband G image (HL),
2048 × 1080 size B image, subband B image (LL),
As described.

  In FIG. 15, the normal Bayer array G image shows a method of performing wavelet transform in units of two adjacent upper and lower lines. As another method, there is a method of performing wavelet transform in units of two adjacent left and right pixels. However, this method is the same as the processing described in the processing of the R image and B image of the double-density Bayer array described in FIG.

Further, in FIG. 13, the pixel centroid position of the 2048 × 1080 size subband image that has passed through the low-pass filter after wavelet transform of the double-density Bayer array has been described. However, in the normal Bayer array, the R image and the B image are As with other inventions, it is a simple thinning process. For this reason, the pixel position is the Bayer array as it is, and the description is omitted.
The above is the details of the processing for the normal Bayer array.

  As described above, it has been described that the compression / expansion I / F unit 105 can correspond to any of the image pickup elements having various pixel arrays as shown in FIG.

  Next, the subband image that is output from the compression / expansion I / F unit 105 has a size of 2048 × 1080 for any of the image sensor units 102 having various pixel arrangements as shown in FIG. It will be described that the processing after the compression / decompression processing unit 106 can be a common processing.

  FIG. 16 summarizes subband images that are output from the compression / expansion I / F unit 105 with respect to the image sensor units 102 having various pixel arrangements described so far.

In particular,
* RGB full pixel method ・ G: 2048 × 1080 size subband (LL / HL / LH / HH) 4 components ・ B: 2048 × 1080 size subband (LL / HL / LH / HH) 4 components ・ R: 2048 × 1080 size subband (LL / HL / LH / HH) 4 components → 2048 × 1080 size subband 12 components

* Double-density Bayer array ・ G: 2048 × 1080 size subband (LL / HL / LH / HH) 4 components ・ B: 2048 × 1080 size subband (LL / HL) 2 components ・ R: 2048 × 1080 size Subband (LL / HL) 2 components → 2048 × 1080 size subband 8 components

* Diagonally arranged 3 plate system-G: 2048 x 1080 size subband (LL / HL) 2 components-B: 2048 x 1080 size subband (LL / HL) 2 components-R: 2048 x 1080 size subband (LL / HL) 2 components → 2048 x 1080 size subbands are 6 components

* Bayer array G: 2048 x 1080 size subband (LL / HL) 2 components B: 2048 x 1080 size subband (LL) 1 component R: 2048 x 1080 size subband (LL) 1 component → The 2048 × 1080 size subbands can be summarized as the difference in the number of subbands of 4 components and 2048 × 1080 size. (In the figure, the non-output subband image is drawn with a broken line.)

  In addition, since each of the RGB images has a 2048 × 1080 size subband (LL) (a subband surrounded by a thick frame in the figure), as described above, simple color separation is possible. Can be confirmed as an image.

  Next, a specific processing method of the compression / expansion I / F unit 105 will be described for image pickup devices having various pixel arrays.

FIGS. 17 and 18 are diagrams for explaining a processing method (FIG. 17 is a process at the time of wavelet transform, and FIG. 18 is a process at the time of inverse transform) when the RGB full pixel method is supported.
17 and 18 are the same as FIG. 9 and FIG. As illustrated in FIG. 17, the compression / decompression I / F unit 105 performs the same processing on all of the R image, the G image, and the B image with respect to the RGB full pixel recording RAW data. At this time, the compression / decompression I / F unit 105 performs wavelet transform in the horizontal and vertical directions of the R image, the G image, and the B image, and outputs 2048 × 1080 size subbands for 12 components. Also, as shown in FIG. 18, the compression / decompression I / F unit 105 performs the inverse transformation to restore the RGB full-pixel reproduction RAW data.

19 and 20 are diagrams for explaining a processing method (FIG. 19 is processing at the time of wavelet transform, and FIG. 20 is processing at the time of inverse conversion) in the case of dealing with an image sensor having a double-density Bayer array.
As shown in FIG. 19, the compression / decompression I / F unit 105 performs wavelet transform on the G image in the recording RAW data in the double-density Bayer array both in the horizontal direction and in the vertical direction. However, after scanning as shown in FIG. 11B is performed on the R image and the B image, only the horizontal wavelet transform is performed, and the vertical wavelet transform circuit performs bypass processing and outputs it as it is. Here, in FIG. 19, the portion to be bypassed is indicated by hatching. In the following FIGS. 20 to 24, a portion for performing the bypass processing is also expressed. As a result, the compression / decompression I / F unit 105 outputs 2048 × 1080 size subbands for eight components. Also, as shown in FIG. 20, the compression / decompression I / F unit 105 bypasses the vertical wavelet synthesis circuit and performs only the horizontal wavelet synthesis for the R image and the B image even in inverse transformation. The G image can be returned to the reproduction RAW data of the double-density Bayer array by performing wavelet synthesis in both the vertical direction and the horizontal direction.

FIGS. 21 and 22 are diagrams for explaining a processing method (FIG. 21 is a process at the time of wavelet transform, and FIG. 22 is a process at the time of inverse transform) in the case of corresponding to the diagonally arranged three-plate method.
As shown in FIG. 21, the compression / decompression I / F unit 105 performs the same processing on all of the R image, G image, and B image in the obliquely arranged three-plate recording RAW data. At this time, the compression / decompression I / F unit 105 performs scanning as shown in FIG. 14B, and then performs only the horizontal wavelet transform of the R image, G image, and B image, and the vertical wavelet transform circuit. Performs bypass processing and outputs as is. As a result, the compression / expansion I / F unit 105 outputs 2048 × 1080 size subbands for six components. Also, as shown in FIG. 22, the compression / decompression I / F unit 105 performs the same processing for all of the R image, G image, and B image even in inverse transformation, bypasses the vertical wavelet synthesis circuit, and performs horizontal processing. Only the wavelet synthesis is performed. Thereby, it is possible to return to the reproduction RAW data of the oblique arrangement of the three plates.

FIG. 23 and FIG. 24 are diagrams for explaining a processing method (FIG. 23 is processing at the time of wavelet transform, and FIG. 24 is processing at the time of inverse conversion) in the case of corresponding to an image sensor with a normal Bayer array.
As shown in FIG. 23, the compression / decompression I / F unit 105 performs the wavelet transform only in the horizontal direction after performing the scan shown in FIG. 15B on the G image in the recording RAW data of the normal Bayer array. However, the R image and the B image are bypassed without performing horizontal wavelet transform. The wavelet transform circuit in the vertical direction performs bypass processing on all of the R image, G image, and B image and outputs them as they are. As a result, 2048 × 1080 size subbands are output for four components.

  Also, as shown in FIG. 24, the compression / decompression I / F unit 105 bypasses the wavelet synthesis circuit in the vertical direction for all of the R image, G image, and B image even in the inverse transformation, and only the G image in the horizontal direction. Perform wavelet synthesis. The R and B images are bypassed without performing horizontal wavelet synthesis. As a result, it is possible to restore the playback RAW data in the normal Bayer array.

  In this way, the compression / decompression I / F unit 105 can deal with image pickup devices having various pixel arrays by performing the bypass processing of the wavelet circuit in accordance with the type of input recording RAW data. is there. Therefore, by controlling only the processing of the compression / decompression I / F unit 105, the processing after the compression / decompression processing unit 106 can be made a common process in which the number of components of the input subband image is different.

  Further, the compression / decompression processing unit 106 and the subsequent processes can be performed in parallel processing. Specifically, in the compression / decompression processing unit 106, a circuit for the subband image D105-LL, a circuit for the subband image D105-HL, a circuit for the subband image D105-LH, and a subband image D105- in FIG. Three HH circuits are provided in parallel for each of RGB (12 systems in total). For this reason, it is possible to use all in the case of a 4K RGB full image and only a part in other cases. As a result, the operation speed can be reduced by the degree of parallelism. That is, a configuration suitable for real-time processing can be taken.

  Conventionally, in a system that handles a high-resolution image such as a 4K image, the amount of signal processing to be handled inevitably increases. Also, from the viewpoint of characteristics such as S / N and sensitivity of the image pickup device, it has been difficult to realize the recording unless RAW data is recorded in advance by using a so-called Bayer array or the like and suppressing the number of pixels. However, since all RGB pixels do not exist, if compression recording is performed with RAW data, the compression efficiency has not increased as described in the Background Art section.

  On the other hand, by using the compression / decompression I / F unit 105 and the compression / decompression processing unit 106 according to the first embodiment described above, the entire screen, which is a feature of the wavelet transform, can be converted. The high compression efficiency can be utilized. In particular, regarding the RAW data compression of the R image and the B image in the double-density Bayer array, it can be expected to greatly improve the compression efficiency by taking advantage of the original characteristics of the wavelet transform.

In addition, it is possible to obtain screens with different resolutions by subband division from one compression code and use these screens. This is because even simple color separation processing is performed by wavelet transformation.
As a result, although it is a low-resolution simple display system, it is possible to construct a very hardware-efficient system that does not require any special color separation circuit or down-conversion circuit.

  Further, the processing of the compression / decompression I / F unit 105 and the compression / decompression processing unit 106 are applicable to any pattern of the conventional RGB full pixel method, double-density Bayer array, three-plate method of diagonal array, and normal Bayer array. It is possible. For this reason, there is an effect that hardware common processing can be performed as “difference in the number of subband image components”. This indicates that, in a system that handles high-resolution images, parallel processing of hardware is possible and real-time processing can be realized.

<2. Second Embodiment>
[Example of compressing and decoding an image using Haar transform]

Next, an example applied to the imaging apparatus 10 according to the second embodiment of the present invention will be described with reference to FIGS.
As described above, the compression / decompression I / F unit 105 provides the most effective compression method by combining with the wavelet transform. However, not only wavelet transformation but also Haar transformation with less hardware burden can be realized. At this time, the compression / decompression I / F unit 105 performs Haar transform on the image data of colors whose pixel positions are alternately shifted for every two adjacent pixels in the diagonal direction as two adjacent upper and lower line units. Become.

  FIG. 25 shows an example in which the compression / decompression I / F unit 105 performs Haar transform on the double-density Bayer array.

FIG. 25A shows a double density Bayer array.
First, the G image will be described. Since the G image has 4K full pixels, the Haar transform may be performed instead of the wavelet transform in the horizontal direction and the vertical direction in the same manner as the processing described in FIG.

At this time, the compression / decompression I / F unit 105 performs the adjacent upper and lower 2-line units for R and B image data out of RGB image data output from the image sensor unit 102 having a double-density Bayer array. The Haar transformation is performed for every two pixels adjacent to each other in the diagonal direction. For the G image data, the horizontal Haar transformation in units of one line of pixels and the vertical direction of Haar in units of one column of pixels. Perform conversion.
The Haar transform is considered as a 2-tap subband filter and is equivalent to taking the sum and difference between two pixels. The sum is LPF and the difference is HPF.
That is, as shown in FIG. 25B, a Haar transform in the horizontal direction is performed for every two pixels on a 4096 × 2160 size G image. As a result, as shown in FIG. 25C, the low-frequency subband G image (L) (sum) and the high-frequency subband G image (H) (difference) having a size of 2048 × 2160 in which the horizontal direction is halved are obtained. .

Next, FIG. 25C shows a 2048 × 2160 low-frequency subband G image (L) (sum) and a high-frequency subband G image (H) (difference) each having a half size in the horizontal direction. Thus, the vertical Haar transformation is performed. As a result, as shown in FIG. 25D, the 2048 × 2160 low-frequency subband G image (L) (sum) and the high-frequency subband G image (H) (difference) are divided into four types of 2048 × 1080-size subbands. It is converted into a band G image.
・ Subband G image (LL) of 2048 × 1080 size that passed through low-pass filter (sum) in both horizontal and vertical directions
・ Sub-band G image (LH) of 2048 × 1080 size through low-pass filter (sum) in the horizontal direction and high-pass filter (difference) in the vertical direction
・ Sub-band G image (HL) of 2048 × 1080 size through high-pass filter (difference) in horizontal direction and low-pass filter (sum) in vertical direction
・ Subband G image (HH) of 2048x1080 size that passed high-pass filter (difference) in both horizontal and vertical directions
As described above, it is converted into four types of 2048 × 1080 size subband G images.

Since the R image and the B image have a checkered pattern, the R image and the B image are subjected to Haar transform instead of wavelet transform in units of two adjacent upper and lower lines.
Specifically, as shown in FIG. 25B, the R pixel performs Haar transformation for every two diagonal pixels adjacent to each other in the upper and lower two lines.

With the above,
2048 × 1080 size subband R image through low-pass filter (sum),
2048 × 1080 size subband R image through high-pass filter (difference),
Convert to
Similarly, the B pixel is also subjected to Haar transform for every two pixels that are adjacent to each other in the upper and lower two line units (in the diagonal direction opposite to R),
2048x1080 size subband B image through low-pass filter (sum),
2048x1080 size subband B image through high-pass filter (difference),
Convert to
Thus, the subband image of 2048 × 1080 size is already obtained, and the size matches that of the G subband image.

This means that the subband images of 2048 × 1080 size have been arranged in RGB, and an RGB full image of 2048 × 1080 size was obtained, and simple so-called color separation was performed. It will be.
As a result, the subband image that is output from the compression / decompression I / F unit 105 has the same size regardless of whether it is a 4K full pixel system or a 4K double density Bayer array. Can be processed. This is the same as in the wavelet transform.

In FIG. 25C and FIG. 25D, the subband image is described as follows for the purpose of explaining that the R image and the B image have been subjected to Haar transform in an oblique direction.
-Subband R image (LL) of 2048x1080 size through low-pass filter (sum)
-Subband R image (HH) of 2048x1080 size through high-pass filter (difference)
-Subband B image (LL) of 2048x1080 size through low-pass filter (sum)
-Subband B image (HH) of 2048x1080 size through high-pass filter (difference)
In addition, the diagonal directions of the R pixel and the B pixel shown here are only examples, and it is clear that if the starting pixel positions of the R pixel and the B pixel are different, the diagonal directions are reversed.

  Next, a case where Haar transformation is performed on the diagonally arranged three-plate method shown in FIG. 2 will be described with reference to FIG.

FIG. 26A shows an array of a diagonal array three-plate system.
In FIG. 26B, as described with reference to FIG. 14, it is considered that there is no pixel marked in gray, so this is the same as the R pixel or B pixel in the double-density Bayer array described with reference to FIG. 25. It can be considered the same.
The difference is that the R image, the B image, and the G image all have the same pattern.
Therefore, it is understood that Haar transform may be performed on R image, B image, and G image instead of wavelet transform in units of two adjacent upper and lower lines.

At this time, the compression / expansion I / F unit 105 applies the two adjacent pixels in the diagonal direction as two adjacent upper and lower line units to all of the RGB image data from the diagonally arranged three-plate image sensor unit 102. Perform Haar transform.
Specifically, as described with reference to FIG. 26B, Haar conversion is performed for every two diagonal pixels adjacent to each other in the upper and lower two lines.

With the above,
2048x1080 size subband image through low-pass filter (sum),
2048x1080 size subband image through high-pass filter (difference),
The RGB images are converted respectively.
From the above, it can be seen that the image is already a 2048 × 1080 size subband image.

This means that the subband images of 2048 × 1080 size are aligned in RGB, and a full RGB image of 2048 × 1080 has been obtained, so that simple so-called color separation has been performed. become.
As a result, the subband image that is output from the compression / decompression I / F unit 105 has the same size in both the 4K full pixel method and the 4K diagonally arranged three-plate method. Common processing can be performed. This is the same as in the wavelet transform.

In FIG. 26C and FIG. 26D, it describes as follows for the purpose of performing the Haar transformation in the oblique direction.
-Subband R image (LL) of 2048x1080 size through low-pass filter (sum)
-Subband R image (HH) of 2048x1080 size through high-pass filter (difference)
・ Subband G image of 2048 × 1080 size that passed through low-pass filter (sum), subband G image (LL)
・ Subband G image of 2048 × 1080 size that passed through high-pass filter (difference), subband G image (HH)
-Subband B image (LL) of 2048x1080 size through low-pass filter (sum)
-Subband B image (HH) of 2048x1080 size through high-pass filter (difference)
In addition, the diagonal direction of the pixel shown here is an example, and it is clear that if the starting pixel position of the R pixel and the B pixel are different, the diagonal direction is reversed.

  Next, a case where Haar transform is performed on the normal Bayer array of FIG. 2 will be described with reference to FIG.

FIG. 27A shows a so-called normal Bayer array.
In FIG. 27B, the positions where the actual pixels do not exist are shown in gray, separated into RGB. The R image and the B image are already decimated images of 2048 × 1080 size in a separated state.
For the G image, as can be seen from the figure, this can be considered the same as the R or B pixel of the double-density Bayer array described with reference to FIG.
Therefore, the G image is subjected to Haar transform instead of wavelet transform in units of two adjacent upper and lower lines.

At this time, the compression / decompression I / F unit 105 does not perform Haar conversion on the R and B image data out of the RGB image data from the Bayer array image sensor unit 102, and applies the G image data to the G image data. Thus, Haar conversion is performed for every two adjacent pixels in the diagonal direction as two adjacent upper and lower line units.
Specifically, as shown in FIG. 27B, the G pixel is subjected to Haar transformation for every two diagonal pixels adjacent to each other in the upper and lower two lines.

With the above,
2048x1080 size sub-band G image through low-pass filter (sum),
2048 × 1080 size subband G image through high-pass filter (difference),
Convert to
From the above, it can be seen that the image is already a 2048 × 1080 size subband image.
This means that the R-band and B-images are thinned out, but the 2048 × 1080-size subband images are aligned in all of the RGB images. For this reason, an RGB full image of 2048 × 1080 is obtained, and simple so-called color separation is performed.
As a result, the subband image that is output from the compression / expansion I / F unit 105 has the same size regardless of whether it is a 4K full pixel system or a 4K normal Bayer array. It can be a process. This is the same as in the wavelet transform.

In FIG. 27C and FIG. 27D, the G image has been subjected to a Haar transformation in an oblique direction.
2048 × 1080 size R image, subband R image (LL),
A 2048 × 1080 size subband G image that has been passed through a low-pass filter (sum) is converted into a subband G image (LL),
A 2048 × 1080 size subband G image that has passed through a high-pass filter (difference) is converted into a subband G image (HH),
2048 × 1080 size B image, subband B image (LL),
It is described as.
In addition, the oblique direction of the G pixel shown here is an example, and it is clear that if the starting pixel position of G is different, the reverse oblique direction is obtained.

According to the second embodiment described above, the compression / decompression I / F unit 105 can be realized not only by the wavelet transform but also by Haar transform with simpler hardware. For this reason, as in the first embodiment described above, the subsequent processing can be performed in parallel. In addition, since color separation can be easily performed, the manufacturing cost of the apparatus can be reduced.
Although the wavelet transform has been described with reference to the JPEG2000 5 × 3 reversible wavelet transform coefficient, it is clear that the wavelet transform can be realized by a 9 × 7 irreversible transform or other transform coefficients. It is not limited to Haar transformation.

<3. Modification>
The processing of the compression / decompression I / F unit 105 and the processing of the compression / decompression processing unit 106 described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, it can be executed by a computer in which a program constituting the software is incorporated in dedicated hardware or a computer in which programs for executing various functions are installed. is there. For example, what is necessary is just to install and run the program which comprises desired software in a general purpose personal computer.

  Further, a recording medium on which a program code of software that realizes the functions of the above-described embodiments may be supplied to the system or apparatus. It goes without saying that the function is also realized by a computer (or a control device such as a CPU) of the system or apparatus reading and executing the program code stored in the recording medium.

  As a recording medium for supplying the program code in this case, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like is used. Can do.

  Further, the functions of the above-described embodiment are realized by executing the program code read by the computer. In addition, based on the instruction of the program code, the OS running on the computer performs part or all of the actual processing. The case where the functions of the above-described embodiment are realized by the processing is also included.

In the above embodiments, the present invention is applied to an image pickup apparatus. However, the present invention is obtained from a double-density Bayer array, an oblique array three-plate method, or a Bayer array image sensor in an apparatus other than the image pickup apparatus. The present invention can also be applied to compression-coding the RAW data to be generated.
In addition, the present invention is a pixel array other than a double-density Bayer array, a diagonal array three-plate system, and a Bayer array, and the pixel positions are shifted alternately in the horizontal direction or the vertical direction for at least one of the three primary colors. The present invention can be similarly applied to compression-coding RAW data obtained from an image sensor having a pixel arrangement.

  Further, the present invention is not limited to the above-described embodiments, and it is needless to say that other various application examples and modifications can be taken without departing from the gist of the present invention described in the claims.

  DESCRIPTION OF SYMBOLS 10 ... Imaging device, 32 ... Wavelet transformation part, 33 ... Quantization part, 35 ... Entropy encoding part, 36 ... Control part, 37 ... Code amount measurement part, 38 ... Entropy decoding part, 39 ... Dequantization part, DESCRIPTION OF SYMBOLS 40 ... Wavelet inverse transformation part, 101 ... Lens block, 102 ... Imaging device part, 103 ... Camera signal processing part, 104 ... Monitor out part, 105 ... Compression / decompression I / F part, 106 ... Compression / decompression processing part, 107 ... Recording Media interface unit 108 ... Recording media unit 109 ... Viewfinder signal processing unit 110 ... Viewfinder unit 111 ... System control unit 112 ... Operation unit D102 ... Recording RAW data D105-LL ... Subband image D105 -HL ... subband image, D105-LH ... subband image, D105-HH ... subband image De image

Claims (6)

Of the image data output from the image sensor having a pixel array in which the pixel positions are alternately shifted in the horizontal direction or the vertical direction for at least one of the three primary colors, an image having a color in which the pixel positions are alternately shifted. When the adjacent upper and lower two lines of pixels are scanned in a zigzag manner for the data, only the horizontal subband conversion is performed, and the subband conversion is not performed for the remaining color image data, or two adjacent left and right columns An image processing apparatus comprising: a subband dividing unit that performs only subband conversion in the vertical direction when the pixels are scanned in a zigzag manner and performs subband division without performing subband conversion on the remaining color image data . The subband division unit performs Haar transform for each pixel adjacent in an oblique direction with two adjacent upper and lower lines as a unit with respect to image data of a color in which the pixel positions are alternately shifted. Image processing apparatus. Of the image data output from the image sensor having a pixel arrangement in which the pixel positions are alternately shifted in the horizontal direction or the vertical direction for at least one of the three primary colors, the subband dividing unit constituting the image processing device, When image data of the color where the pixel positions are alternately shifted is scanned zigzag on the adjacent upper and lower two lines of pixels, only the horizontal subband conversion is performed and the image data of the remaining color is sub Image processing having a step of performing only subband conversion in the vertical direction and performing no subband conversion on the image data of the remaining colors when band conversion is not performed, or when two adjacent pixels on the left and right are scanned in a zigzag manner Method. Of the image data output from the image sensor having a pixel array in which the pixel positions are alternately shifted in the horizontal direction or the vertical direction for at least one of the three primary colors, an image having a color in which the pixel positions are alternately shifted. When the adjacent upper and lower two lines of pixels are scanned in a zigzag manner for the data, only the horizontal subband conversion is performed, and the subband conversion is not performed for the remaining color image data, or two adjacent left and right columns A sub-band splitting unit that performs only vertical sub-band conversion and does not perform sub-band conversion on the remaining color image data,
A compression encoding unit that performs processing for compressing and encoding image data output from the subband splitting unit in parallel for each of the bands divided by the subband splitting unit and for each of the three primary colors. Image processing device. Of the image data output from the image sensor having a pixel arrangement in which the pixel positions are alternately shifted in the horizontal direction or the vertical direction for at least one of the three primary colors, the subband dividing unit constituting the image processing device, When image data of the color where the pixel positions are alternately shifted is scanned zigzag on the adjacent upper and lower two lines of pixels, only the horizontal subband conversion is performed and the image data of the remaining color is sub When band conversion is not performed or when two adjacent pixels on the left and right are scanned in a zigzag manner, only subband conversion in the vertical direction is performed, and subband division is performed for the remaining color image data without performing subband conversion. Steps,
An image processing method comprising: a step in which the subband dividing unit performs a process of compressing and encoding image data subjected to subband division in parallel for each band to be divided and for each of the three primary colors. An image sensor having a pixel array in which pixel positions are alternately shifted in the horizontal direction or the vertical direction for at least one of the three primary colors;
Of the image data output from the image sensor, only horizontal sub-band conversion is performed when the adjacent upper and lower two lines of pixels are scanned in a zigzag manner for the color image data in which the pixel positions are alternately shifted. If the image data of the remaining color is not subjected to subband conversion, or if two adjacent pixels on the left and right are scanned in a zigzag manner, subband division is performed to perform only the subband conversion in the vertical direction. A subband dividing unit that does not perform subband conversion on the image data of
A compression encoding unit that performs the process of compressing and encoding the image data output from the subband splitting unit in parallel for each band divided by the subband splitting unit and for each of the three primary colors;
An image pickup apparatus comprising: a recording unit that records image data compressed and encoded by the compression encoding unit.