JP2017085556A - Image processing apparatus, imaging apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for enabling efficient encoding of RAW data.SOLUTION: An image processing apparatus for encoding image data of a Bayer array, comprises: color separation means 121 that separates a plurality of color components constructing image data into a plurality of plane image data on the basis of a Bayer array; sum plane generation means 124 that generates plane image data of a sum plane by performing addition of components having the same color among from the plurality of plane image data; difference plane generation means 125 that generates plane image data of a difference plane by performing subtraction of components having the same color; and encoding means for performing compression encoding of plane image data that performs compression encoding of plane image data of a sum plane and a difference plane.SELECTED DRAWING: Figure 1B

Description

The present invention relates to an image processing device, an imaging device, an image processing method, and a program.

  Recent imaging apparatuses such as digital cameras and digital camcorders employ CCD sensors or CMOS sensors as imaging elements. These sensors are mainly composed of a single plate type, and a pixel value of one color component (for example, one of RGB) is obtained by one pixel by a color filter array (hereinafter referred to as CFA) on the sensor surface. get. By using CFA, for example, image data of a Bayer array (R) (red), G0 (green), B (blue), and G1 (green) arranged in a periodic pattern as shown in FIG. (Hereinafter referred to as RAW data). Since human vision has high sensitivity to the luminance component, the RAW data in FIG. 2 (a) is assigned twice as many pixels as the green component containing the luminance component compared to the red and blue components. ing. Since RAW data has only one color component information per pixel as described above, it is necessary to generate red, blue, and green pixel values for one pixel using demosaic processing. The “de-mosaic” process is a process of creating a full-color image by complementing color information by collecting and giving missing color information from surrounding pixels to each pixel.

  In general, an imaging apparatus encodes and records image data of an RGB signal obtained by demosaicing or a YUV signal obtained by conversion from an RGB signal. On the other hand, in recent years, a method of encoding and recording RAW data before demosaicing has been proposed.

  For example, Patent Document 1 discloses a method of encoding after classifying RAW data into signal components (R, G0, B, G1) and arranging them into four components (planes).

JP 2003-125209 A

  However, by separating the RAW data for each color signal component, the G0 component and G1 component that are originally close to each other and have the same color and high correlation are separated into different planes. For this reason, the pixel values of adjacent pixels may be separated in G0 and G1, respectively, and as a result, the encoding efficiency may be deteriorated.

Therefore, an object of the present invention is to provide a technique that enables efficient encoding of RAW data.

The present invention for solving the above problems is an image processing apparatus that encodes Bayer array image data,
Color separation means for separating a plurality of color components constituting the image data into a plurality of plane image data based on the Bayer array;
A sum plane generating means for generating a sum plane plane image data by adding components of the same color among the plurality of plane image data;
Difference plane generating means for subtracting the same color components to generate plane image data of a difference plane;
Coding means for compressing and coding the plain image data, the plain image data of the same color component comprising coding means for compressing and coding the plain image data of the sum plane and the difference plane It is characterized by.

  According to the present invention, it is possible to efficiently encode RAW data.

It is a block diagram which shows the structural example of the imaging device which concerns on embodiment of invention. It is a block diagram which shows the structural example of the image process part which concerns on embodiment of invention. (A) The figure which shows a Bayer arrangement | sequence, (b) The figure which shows a double-density Bayer arrangement | sequence. It is a flowchart which shows an example of the encoding operation | movement which concerns on Embodiment 1 of invention. It is a figure which shows typically the color separation of RAW data of a Bayer arrangement | sequence, and the plane formation method. It is explanatory drawing which shows an example of the sum plane production | generation method which concerns on embodiment of invention. It is explanatory drawing which shows an example of the difference plane production | generation method which concerns on embodiment of invention. It is a figure which illustrates the subband image after wavelet transform. It is a figure showing handling of the pixel value for every color component of a double density Bayer. It is a flowchart which shows an example of the encoding operation | movement which concerns on Embodiment 2 of invention. It is a flowchart which shows an example of the encoding operation | movement which concerns on Embodiment 3 of invention. It is a figure which shows an example of weighting of the target code amount of a sum plane and a difference plane for every subband which concerns on Embodiment 3 of invention. It is a block diagram which shows the structural example of the image process part which concerns on Embodiment 4 of invention. It is a flowchart which shows an example of the encoding operation | movement which concerns on Embodiment 4 of invention. It is a figure which shows an example of the weighting of the target code amount for every plane and subband which concerns on Embodiment 5 of invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the following embodiments. That is, the following embodiments do not limit the invention according to the claims, and all combinations of features described in the embodiments are not necessarily essential to the solution means of the invention.

  In addition, each functional block described in the present embodiment does not necessarily need to be individual hardware. That is, for example, the functions of some functional blocks may be executed by one piece of hardware. In addition, the function of one functional block or the functions of a plurality of functional blocks may be executed by some hardware linked operations. Further, the function of each functional block may be executed by a computer program developed on the memory by the CPU.

[Embodiment 1]
Hereinafter, a first embodiment of the invention will be described. In the present embodiment, an imaging apparatus such as a digital camera will be described as an example. However, any apparatus may be used as long as it can compress Bayer array RAW image data. For example, it may be a communication device such as a mobile phone, a smartphone, a tablet information terminal, a notebook information terminal, a computer, an information processing device, or the like. Hereinafter, an imaging device corresponding to an embodiment of the invention will be described.

  First, with reference to FIG. 1A, a main configuration of the imaging apparatus 100 of the present embodiment will be described. In FIG. 1, the control unit 101 includes, for example, a CPU (MPU), a memory (DRAM, SRAM), a nonvolatile memory (EEPROM), and the like, and executes various processes (programs) to control each block of the imaging apparatus 100. And control data transfer between each block. Further, the control unit 101 controls each block of the imaging apparatus 100 in accordance with an operation signal from the operation unit 102 that receives an operation from the user.

  The operation unit 102 includes switches for inputting various operations related to shooting such as a power button, a still image recording button, a moving image recording start button, a zoom adjustment button, and an autofocus button. Further, it includes a menu display button, a determination button, other cursor keys, a pointing device, a touch panel, and the like. When these keys and buttons are operated by the user, an operation signal is transmitted to the control unit 101. Note that the imaging apparatus 100 has an electrically erasable / recordable non-volatile memory (not shown), and stores constants, programs, and the like for the operation of the control unit 101.

  The imaging unit 103 controls the light amount of an optical image (subject image) of a subject captured by a lens by an aperture, converts it into an image signal by an imaging element such as a CCD sensor or a CMOS sensor, and performs analog-digital conversion. And transmitted to the image processing unit 104. The image processing unit 104 processes the input digital image signal, and the processed image data is stored in a non-illustrated nonvolatile memory by the control unit 101. Specific processing will be described later. Here, the image signal obtained by the image sensor is so-called RAW image data, which is RGB data in a Bayer array.

  The audio input unit 105 collects (collects) sound around the imaging apparatus 100 using, for example, a built-in omnidirectional microphone or an external microphone connected via an audio input terminal, and performs analog-digital conversion. And transmitted to the voice processing unit 106. The sound processing unit 106 performs processing related to sound such as level optimization processing of the input digital sound signal. The audio data processed by the audio processing unit 106 is stored in a non-illustrated non-volatile memory by the control unit 101.

  Image data and audio data processed by the image processing unit 104 and the audio processing unit 106 and stored in a nonvolatile memory (not shown) are transmitted to the recording / reproducing unit 110 by the control unit 101 and recorded on the recording medium 111. . Here, the recording medium 111 may be a recording medium built in the imaging apparatus or a removable recording medium, and records a compressed image signal, a compressed audio signal, an audio signal, various data, and the like generated by the imaging apparatus 100. If you can. For example, the recording medium 111 includes all types of recording media such as a hard disk, an optical disk, a magneto-optical disk, a CD-R, a DVD-R, a magnetic tape, a nonvolatile semiconductor memory, and a flash memory.

  Also, the compressed image signal and the compressed audio signal recorded on the recording medium 111 are decoded by the image processing unit 104 and the audio processing unit 106, the image is displayed on the display unit 108 via the display control unit 109, and the audio is transmitted from the speaker. Audio is output from the unit 107.

  The communication unit 112 performs communication between the imaging device 100 and an external device. For example, the communication unit 112 transmits or receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal. Also, it transmits and receives control signals related to shooting, such as shooting start and end commands, and other information. The communication unit 112 is, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a Wireless USB, or a GPS receiver.

  Next, the configuration and processing flow of the image processing unit 104 corresponding to the present embodiment will be described with reference to the block diagram of FIG.

  The image processing unit 104 according to the present embodiment functions as an encoding unit, and includes a color separation unit 121, a memory I / F unit 122, a memory 123, a sum plane generation unit 124, a difference plane generation unit 125, an encoding order setting unit 126, The code amount control unit 127, the wavelet transform unit 128, the quantization unit 129, the entropy encoding unit 130, and the code amount determination unit 131 are configured.

  The color separation unit 121 separates the Bayer array RAW data into independent plane image data (hereinafter simply referred to as “plane”) for each of R, G0, G1, and B pixels as shown in FIG. Among the planes of the respective color components generated by the color separation unit 121, the R and B planes are written to the memory 123 via the memory I / F unit 122, and the G0 and G1 planes are the sum plane generation unit 124 and the difference plane generation. Input to the unit 125. The memory I / F unit 122 arbitrates memory access requests from the processing units and performs read / write control on the memory 123. The memory 123 is a storage area for holding various data output from each processing unit.

  As shown in FIG. 5, the sum plane generation unit 124 adds the pixel values at the same position of the two input planes to obtain the pixel value of the output sum plane. The sum plane generated by the sum plane generation unit 124 is written into the memory 123 via the memory I / F unit 122. As illustrated in FIG. 6, the difference plane generation unit 125 subtracts the pixel values at the same position of the two input planes to obtain the pixel value of the output difference plane. The difference plane generated by the difference plane generation unit 125 is written into the memory 123 via the memory I / F unit 122.

  The encoding order setting unit 126 sets the encoding order of the R, B, sum, and difference planes, and controls the memory I / F unit 122 to read the planes in the order set from the memory 123. . The code amount control unit 127 sets the target code amount of the encoded data of each plane in the quantization unit 129 and the code amount determination unit 131.

  The wavelet transform unit 128 performs wavelet transform on the R, B, sum, and difference planes read from the memory 123 via the memory I / F unit 122, and then generates transform coefficients generated in the subband image format. Is sent to the quantization unit 129. Here, the wavelet transform is a conversion process for converting an image to be processed into a hierarchical image (subband) for each frequency band. Specifically, it refers to a process of separating a low-frequency component and a high-frequency component by applying low-pass filter processing and high-pass filter processing to the entire image. The transform coefficient after the wavelet transform takes the form of a subband image as shown in FIG. The wavelet transform repeatedly divides only the low frequency sub-band every time the number of divisions is increased. This is because the encoding efficiency is increased by utilizing the characteristic that much of the image energy is concentrated in the low frequency band. Used for.

  Next, the subband image will be described with reference to FIG. FIG. 7 is an example of a subband image generated by executing the wavelet transform twice, and shows that it is formed by a total of seven subbands. Here, L represents a low frequency, H represents a high frequency, and numbers before L and H represent hierarchical levels. For example, 1HL represents a subband image with a layer level = 1 where the horizontal direction is a high-frequency component and the vertical direction is a low-frequency component. Also, the upper left subband has more low-frequency components and the image energy is concentrated.

  The quantization unit 129 performs quantization on the subband image sent from the wavelet transform unit 128 and sends the result to the entropy coding unit 130. Note that the parameters used for quantization are set based on the target code amount set by the code amount control unit 127. The entropy encoding unit 130 entropy encodes the subband image quantized by the quantization unit 129 to generate encoded data. The generated encoded data is sent to the code amount determination unit 131. As entropy coding executed by the entropy coding unit 130, for example, EBCOT (Embedded Block Coding with Optimized Truncation) employed in the JPEG2000 system is used. The code amount determination unit 131 determines whether the encoded data sent from the entropy encoding unit 130 is larger than the target code amount set by the code amount control unit 127.

  In the present embodiment, when the encoded data is larger than the target code amount, the code amount determination unit 131 further compresses the encoded data so that the size of the encoded data satisfies the target code amount. As a method for further compressing the encoded data, for example, post-quantization which is one of the functions of JPEG2000 is used. Post-quantization uses the entropy coding EBCOT used in JPEG2000 to encode the transform coefficient in each bit plane, and truncates the completed encoded data from the lower order bits. How to achieve quantity.

  Next, the encoding process for one frame of RAW data according to the present embodiment will be described based on the flowchart shown in FIG. The flowchart in FIG. 3 can be realized by, for example, a processor functioning as the image processing unit 104 developing and executing a program stored in a memory (ROM) in a work memory (RAM) (another embodiment). The same applies to the flowchart corresponding to FIG.

  First, in S301, the color separation unit 121 separates R, B, G0, and G1 from the RAW data to form a plane for each. After forming a plane for each color component, a sum plane and a difference plane are generated in S302. Specifically, the sum plane generating unit 124 adds the G0 plane and the G1 plane to generate a sum plane. In addition, the difference plane generation unit 125 performs subtraction of the G0 plane and the G1 plane to generate a difference plane. The R and B planes and the sum and difference planes are generated and then written to the memory 123 via the memory I / F unit 122.

  After generating the R, B, sum, and difference planes, the encoding order setting unit 126 sets from which plane the encoding process is executed in S303. Here, as an example, the order of R, B, sum, and difference is set, but the order is not limited. After setting the encoding target plane, the code amount control unit 127 sets a target code amount for the selected plane in S304. In this embodiment, the code amount control unit 127 sets a value obtained by equally dividing the target code amount of the entire RAW data by the number of planes as the target code amount for each plane. That is, the target code amounts of the R, B, sum, and difference planes are set to substantially the same code amount.

  After setting the target code amount, in S305, the wavelet transform unit 128 reads the selected plane from the memory 123 via the memory I / F unit 122, and executes wavelet transform. Next, in S306, the quantization unit 129 quantizes the wavelet-transformed image. The quantization parameter used for quantization is determined based on the target code amount set by the code amount control unit 127.

  After quantization, in S307, the entropy encoding unit 130 compresses and encodes the subband image and outputs encoded data. After the completion of the compression encoding, the code amount determination unit 131 determines whether the code amount of the output encoded data is larger than the target code amount set by the code amount control unit 127 in S308. If it is larger than the target code amount, the process returns to S307, and the entropy encoding unit 130 further compresses the encoded data to satisfy the target code amount. If the code amount is less than or equal to the target code amount, in S309, it is determined whether encoding has been performed for all of the R, B, sum, and difference planes. If there is an unprocessed plane, the process returns to S303 for processing. Continue. On the other hand, when the process is completed for all, the encoding process is terminated.

  In the present embodiment, an example of processing in units of one frame has been described. However, the processing may be performed independently by dividing the frame into arbitrary sizes. In the present embodiment, an example using wavelet transform is shown, but discrete cosine transform used in encoding techniques such as JPEG, JPEG XR, and MPEG-4 AVC may be used instead. Furthermore, the entropy coding may be Huffman coding.

  In this embodiment, pixel data having a Bayer array as shown in FIG. 2A is used. However, a double-density Bayer array as shown in FIG. 2B may be used. The double-density Bayer array is a pixel array in which the pixel density of the Bayer array is doubled and arranged at an angle of 45 °, and is composed of R0, R1, B0, B1, and G. In the double-density Bayer array, the number of G pixels is twice that of R (the total number of R0 and R1) and B (the total number of B0 and B1), as in the Bayer array. When encoding a double-density Bayer array, planes are generated with R0, R1, B0, B1, and G as shown in FIGS. 8A to 8C, and R0 and R1, and B0 and B1, respectively. By generating the sum plane and the difference plane, the encoding process for each plane can be performed in the same manner as the Bayer array.

  In this case, in the process of FIG. 3, a plane is generated with R0, R1, B0, B1, and G in S301, and a sum plane and a difference plane are generated with R0 and R1, and B0 and B1, respectively. Thereafter, the processing from S303 to S309 is executed for the G plane and the R sum plane, the difference plane, the B sum plane, and the difference plane.

  As described above, according to the present embodiment, each color component of the Bayer array is separated into planes that are independent images, and a sum plane is generated from the planes of the same color components by addition, and a difference plane is generated by subtraction. . That is, in the normal Bayer arrangement, the sum plane and the difference plane are generated from the same green color components of G0 and G1, and in the double density Bayer arrangement, the red color components of R0 and R1, and the blue color components of B0 and B1 are generated from each other. A sum plane and a difference plane are generated respectively. Here, since the addition is a low-pass filter operation, the generation of the sum plane corresponds to a process of extracting a low-frequency component with a low-pass filter for the green component of the RAW data. Since subtraction is a high-pass filter operation, the generation of the difference plane corresponds to a process of extracting a high-frequency component with a high-pass filter for the green component of the RAW data. Therefore, the low frequency component and the high frequency component of the green component of the RAW data are separated before encoding processing. As a result, it is possible to reduce the deterioration of the frequency component division efficiency at the time of wavelet transform and increase the compression efficiency of the encoding process. Note that this can also improve the compression efficiency of the encoding process in the discrete cosine transform.

  In this embodiment, the basic target code amount is evenly allocated to R, B, G0, and G1, but the R and B code amounts are reduced and allocated to G0 and G1 for each color component. You may change the ratio of the code amount to allocate.

[Embodiment 2]
Subsequently, an encoding method corresponding to the second embodiment of the invention will be described. In the present embodiment, the configuration of the imaging apparatus 100 and the image processing unit 104 that functions as an encoding unit is the same as that illustrated in FIGS. 1A and 1B of the first embodiment. In the present embodiment, considering that the sum plane corresponds to the low frequency component of the green component of the RAW data and the difference plane corresponds to the high frequency component described in the first embodiment, the code amount of the sum plane is changed to the code amount of the difference plane. It is characterized in that it is set larger.

  Hereinafter, the encoding operation of this embodiment will be described with reference to the flowchart of FIG. In addition, the same code | symbol is attached | subjected and shown to the process similar to Embodiment 1, and description here is simplified. First, in the same manner as in the first embodiment, in S301 to S303, a plane is formed for each of R, B, G0, and G1, a sum plane and a difference plane are generated using the G0 and G1 planes, and then R, B, sum, and difference are generated. It is set to which plane of the plane the encoding process is executed. Also in this embodiment, it shall be set to the order of R, B, the sum, and a difference as an example. Subsequently, in S901, the encoding order setting unit 126 determines whether or not the sum plane is set as an encoding target. When the sum plane is an encoding target, the code amount control unit 127 sets the weighting of the target code amount so that the code amount is larger than that of the difference plane in S902.

  In this embodiment, as a method of weighting the target code amount, a basic target code amount separately set by the code amount control unit 127 (as an example, a value obtained by equally dividing the target code amount of the entire RAW data by the number of planes) Is increased by an arbitrarily set parameter X (%). As a method for setting the target code amount of the sum plane based on weighting, for example, there is a method based on Equation 1. Here, the basic target code amount is T, and the weighted target code amount is T ′.

T ′ = (100 + X) T / 100 (Formula 1)
For example, when X is set to 15, the target code amount of the sum plane is increased by 15% with respect to the basic target code amount. However, increasing the target code amount of the sum plane increases the target code amount of the entire RAW data. Therefore, in the present embodiment, the target code amount of the difference plane is decreased by the amount increased for the sum plane. Specifically, in step S903, the encoding order setting unit 126 determines whether the difference plane is set as an encoding target. When the difference plane is an encoding target, in step S904, the code amount control unit 127 sets weighting so that the target code amount of the difference plane is smaller by X% than the basic target code amount. As a setting method of the target code amount of the difference plane based on weighting, for example, there is a method based on Expression 2. Here, the basic target code amount is T, and the weighted target code amount is T ′.

T ′ = (100−X) T / 100 (Formula 2)
In this way, the increase in the code amount of the entire RAW data can be suppressed by weighting the target code amount of the difference plane so that the code amount of the difference plane decreases as much as the code amount of the sum plane increases. it can. When the encoding target plane set by the encoding order setting unit 126 is neither a sum plane nor a difference plane (that is, R and B planes (both “NO” in S901 and S903)), the code amount control unit 127 The basic target code amount is set as it is. Therefore, no weighting is set. After the code amount control unit 127 sets the target code amount reflecting the weight for each plane in S304, the processing from S305 to S309 is performed as in the first embodiment.

  In the present embodiment, an example of processing in units of one frame has been described. However, the processing may be performed independently by dividing the frame into arbitrary sizes. In the present embodiment, an example using wavelet transform is shown, but discrete cosine transform used in encoding techniques such as JPEG, JPEG XR, and MPEG-4 AVC may be used instead. Furthermore, the entropy coding may be Huffman coding.

  In this embodiment, pixel data having a Bayer array as shown in FIG. 2A is used. However, a double-density Bayer array as shown in FIGS. 2B and 8 may be used. In this case, when a sum plane and a difference plane are generated by R0 and R1, and B0 and B1, respectively, the target code amount of the sum plane can be set larger than the difference plane.

  As described above, according to the present embodiment, the sum plane is more important than the other planes, and by assigning a larger amount of code, it is possible to suppress the deterioration of the low frequency component of the green component of the RAW data. The low frequency component of the image has a large influence on the image quality, and the luminance component of the image contains a green component more than red and blue components. Therefore, by applying the present embodiment, it is possible to maintain the green component of the RAW data, and thus the low-frequency component of the luminance of the image, and suppress image quality deterioration. In this embodiment, the basic target code amount is evenly allocated to R, B, G0, and G1, but the R and B code amounts are reduced and the code amount is allocated to G0 and G1, for each color component. You may change the ratio of the code amount to allocate.

[Embodiment 3]
Subsequently, an encoding method corresponding to the third embodiment of the invention will be described. In the present embodiment, the configuration of the imaging apparatus 100 and the image processing unit 104 that functions as an encoding unit is the same as that illustrated in FIGS. 1A and 1B of the first embodiment. In the present embodiment, a technique using the characteristics of wavelet transform will be described for the weighting of the target code amounts of the sum plane and the difference plane described in the second embodiment.

  Hereinafter, the encoding operation of this embodiment will be described with reference to the flowchart of FIG. In addition, the same code | symbol is attached | subjected and shown to the process similar to Embodiment 1, and description here is simplified.

  First, in the same manner as in the first embodiment, in S301 to S303, a plane is formed for each of R, B, G0, and G1, a sum plane and a difference plane are generated using the G0 and G1 planes, and then R, B, sum, and difference are generated. It is set to which plane of the plane the encoding process is executed. Also in this embodiment, it shall be set to the order of R, B, the sum, and a difference as an example. Subsequently, in S1001, the encoding order setting unit 126 determines whether or not the sum plane is set as an encoding target plane. When the sum plane is an encoding target, the code amount control unit 127 weights the target code amount in step S1002 for the sum plane. Also, the encoding order setting unit 126 determines whether or not the difference plane is set as the encoding target plane. When the difference plane is an encoding target, the code amount control unit 127 weights the target code amount in S1004, which is set for the difference plane.

  In the present embodiment, the code amount control unit 127 sets the target code amount of the sum plane and the difference plane for each subband by using the fact that the frequency components are separated for each subband by the wavelet transform. Details of the target code amount setting method will be described later with reference to FIG. When the encoding target plane set by the encoding order setting unit 126 is neither a sum nor a difference plane (R, B plane), the basic target code amount (for example, the target code amount of the entire RAW data is set to the plane). Set the value as it is (divided equally by number). After the code amount control unit 127 sets the target code amount reflecting the weight for each subband in S304, the processing from S305 to S309 is performed as in the first embodiment.

  Next, the weighting of the target code amount of the sum plane and the difference plane for each subband in this embodiment will be described. An example of the weighting of the target code amount of the sum plane and the difference plane is shown in FIG. In order to simplify the description of the subband image in FIG. 11, the number of divisions by wavelet transform is set to one. Also, the objects at the four corners in FIG. 11 represent the ratio of the code amount of the sum plane and the difference plane of each subband. Since the sum plane corresponds to the low band of the green component of the RAW data, as a result of the wavelet transform, more image energy is concentrated in the low band subbands. Therefore, by making the ratio of the code amount of the sum plane in the low-frequency subband larger than that of the difference plane, more image energy can be left and image quality deterioration can be suppressed.

  On the other hand, it is also assumed that the outline of the image is blurred by giving priority to low-frequency information. Therefore, since the difference plane corresponds to the high frequency of the green component of the RAW data, the high frequency subband is set to a high frequency by making the ratio of the code amount of the difference plane equal to or larger than the sum plane. It is possible to keep the contour of the image while leaving the information. When allocating the code amounts of the sum plane and the difference plane, the sum of the target code amounts of the sum plane and the difference plane is set for each subband. In the present embodiment, a value obtained by further dividing the target code amount for two planes by the number of subbands calculated from the target code amount for the entire RAW data is set as the sum of the target code amounts for the sum plane and the difference plane. . For example, if the target code amount of the entire RAW data is 800 KB and the number of subbands is four, the target code amount for two planes is 400 KB. Further, the sum of the code amount of the sum plane and the difference plane of each subband is set to 100 KB.

  When the amount of code for each subband of the sum plane and difference plane is allocated, the sum of the amount of codes for the sum plane and difference plane for each subband is 100 KB, the number of subbands is four, and the ratio shown in FIG. 11 is used. This will be shown in detail. Since the ratio of the sum plane and the difference plane is 8: 2, the 0LL band is allocated 80 KB for the sum plane and 20 KB for the difference plane. Similarly, since the ratio of the sum plane and the difference plane in the 1HL band and 1LH band is 6: 4, 60 KB is allocated to the sum plane and 40 KB is allocated to the difference plane. In the 1HH band, since the ratio of the sum plane and the difference plane is 4: 6, 40 KB is allocated to the sum plane and 60 KB is allocated to the difference plane.

  In this way, the code amounts of the sum plane and the difference plane are allocated for each subband. It should be noted that the code amount control unit 127 executes the code amount setting for each subband of the sum plane and the difference plane. In the present embodiment, an example of processing in units of one frame has been described. However, the processing may be performed independently by dividing the frame into arbitrary sizes.

  According to the present embodiment described above, the code amounts of the sum plane and the difference plane can be set finely for each subband. Specifically, the code amount of the sum plane corresponding to the low frequency of the green component of the RAW data is large in the low frequency subband, and the difference plane corresponding to the high frequency of the green component of the RAW data in the high frequency subband. Each target code amount is weighted so as to increase the code amount. As an effect, not only a low frequency component but also a high frequency component can be left, and image quality deterioration can be suppressed.

  In this embodiment, the sum of the code amounts of the sum plane and the difference plane of each subband is made equal, but a different value may be set for each subband. For example, instead of 100 KB for 0LL, 1HL, 1LH, and 1HH, it may be 130 KB for 0LL, 100 KB for 1HL, 100 KB for 1LH, and 70 KB for 1HH.

[Embodiment 4]
Subsequently, an encoding method corresponding to the fourth embodiment of the invention will be described. In this embodiment, when weighting the target code amount of the sum plane and the difference plane, the generated code amount of the difference plane is predicted from the variation in the difference value between the G0 plane and the G1 plane constituting the difference plane, and dynamically weighted. To change. In the present embodiment, the configuration of the imaging apparatus 100 is the same as that shown in FIG. 1A of the first embodiment. However, the configuration of the image processing unit 104 as an encoding unit according to the present embodiment is as shown in FIG. The configuration of the image processing unit 104 of this embodiment is different from the configuration of FIG. 1B in that it has an image feature analysis unit 1201 and a weighting setting unit 1202 and performs image feature analysis processing and weighting setting processing. Components similar to those in FIG. 1B are denoted by the same reference numerals, and description thereof is omitted here.

  Hereinafter, the encoding operation of this embodiment will be described with reference to the flowchart of FIG. In addition, the same code | symbol is attached | subjected and shown to the process similar to Embodiment 1, and description here is simplified.

First, similarly to the first embodiment, a plane is formed for each of R, B, G0, and G1 in S301 and S302, and a sum plane and a difference plane are generated using the G0 and G1 planes. Thereafter, in S1301, the image feature analysis unit 1201 analyzes the variation in the pixel value of the difference plane, and the weight setting unit 1202 sets the parameter Y used for the weight setting of the code amount of the sum plane and the difference plane based on the analysis information. In the present embodiment, dispersion based on the expression of Expression 3 is used in analyzing the variation. The variance is δ, m is the total number of components constituting the difference plane, x ^ is the average value of all components of the difference plane, and xi is the value of the i-th component among the 1st to m-th components constituting the difference plane. Represent each.

  The weighting setting unit 1202 has a plurality of threshold values, and sets the parameter Y by comparing the variance output from the image feature analysis unit 1201 with the threshold values. A specific example of the Y setting method will be described later. After setting the parameter Y, in S303, the encoding order setting unit 126 sets which of the R, B, sum, and difference planes is to be encoded. In the present embodiment, as an example, the sum, difference, R, and B are set in this order. In S1302, the encoding order setting unit 126 determines whether the sum plane is set as an encoding target. When the sum plane is set as an encoding target, the code amount control unit 127 sets weighting of the code amount of the sum plane based on the parameter Y in S1303. If the sum plane is not set as an encoding target, the encoding order setting unit 126 determines whether or not the difference plane is set as an encoding target in S1304. When the difference plane is set as an encoding target, the code amount control unit 127 sets the weighting of the code amount of the difference plane based on the parameter Y in S1305. When the encoding target plane set by the encoding order setting unit 126 is neither the sum nor the difference plane (R, B plane), the basic target code amount is set as it is. Here, a value obtained by equally dividing the target code amount of the entire RAW data by the number of planes can be used. After that, after the code amount control unit 127 sets a target code amount reflecting the weight for each plane, the processing from S305 to S309 is performed as in the first embodiment.

  Subsequently, analysis of variance of the difference plane and weighting of the code amount will be described. As described above, the weighting setting unit 1202 has a plurality of threshold values, and sets the weighting parameter Y by comparing the variance output from the image feature analysis unit 1201 with the threshold values. Here, as an example, the threshold values are T1, T2 (T1 <T2), and the weight setting parameter Y is 0, 1, 2. When the variance δ of the difference plane is T1 or less (T1 ≧ δ), Y = 0. When the variance is a value between T1 and T2 (T1 <δ <T2), Y = 1 is set. When the dispersion is equal to or greater than T2 (T2 ≦ δ), Y = 2. When the variance δ of the difference plane is large, the generated code amount of the difference plane increases. Therefore, when Y = 2, which is set when the variance is large, weighting is set so as not to reduce (increase) the code amount of the difference plane as much as possible. Conversely, the smaller the variance, the smaller the generated code amount of the difference plane. Therefore, when Y = 0, which is set when the variance is small, weighting is set so as to increase the code amount of the sum plane as much as possible.

  In addition, a basic target code amount is separately set by the code amount control unit 127 as the target code amount before weighting. Here, a value obtained by equally dividing the target code amount of the entire RAW data by the number of planes is used. The code amount weighting is set such that the sum plane code amount is larger than the difference plane code amount. As an example of the weighting of the code amount, when Y = 0, the target code amount of the sum plane is increased by 10%, and the target code amount of the difference plane is decreased by 10%. Further, when Y = 1, the target code amount of the sum plane is decreased by 5%, the target code amount of the difference plane is increased by 5%, and when Y = 2, the target code amount of the sum plane is decreased by 20%. Is increased by 20%. The ratio of the target code amount shown here is merely an example. The larger the variance of the difference plane, the more target code amount is allocated to the difference plane, and the smaller the variance is, the more target code amount is allocated to the sum plane. Just swing.

  In the present embodiment, an example of processing in units of one frame has been shown, but the above processing may be executed independently by dividing into regions of an arbitrary size. In the present embodiment, an example using wavelet transform is shown, but discrete cosine transform used in coding techniques such as JPEG, JPEG XR, MPEG-4 AVC, etc. may be used instead. Furthermore, the entropy coding may be Huffman coding.

  According to the present embodiment, the target code amount for the high frequency component is appropriately set based on the characteristics of the image while the target code amount for the low frequency component is allocated more than the target code amount for the high frequency component. The image quality can be improved by increasing or decreasing. In the case of an image having a clear outline, since the difference in pixel values between adjacent pixels is larger than that in a picture with a blurred outline, the variance of the difference plane is increased. Therefore, when the variance of the difference plane is large, the outline of the image can be maintained by assigning a larger target code amount to the difference plane, that is, the high frequency component. Further, in the case of a flat image with a small difference between adjacent pixels such as a sky pattern, the variance of the difference plane is reduced. Therefore, when the variance of the difference plane is small, the image quality can be improved by reducing the target code amount of the difference plane and increasing the sum plane, that is, the code amount of the low frequency component.

  In this embodiment, the image feature analysis process is performed on the difference plane. However, the image feature analysis process may be performed on the sum plane or the R and B planes. Even in that case, the difference plane is determined by determining which component is higher in the high frequency component and the low frequency component based on the result of the analysis process, in other words, which component should be assigned the more code amount. It is possible to determine which of the sum plane and the code amount is to be allocated more. For example, when the variance in the sum plane is calculated, it can be determined that the image having a higher flatness such as a sky pattern has a smaller variance in the sum plane, and the flatness is high, that is, the code amount for the sum plane should be increased. On the contrary, when the variance of the sum plane is large, it can be determined that the code amount for the difference plane should be increased.

[Embodiment 5]
Subsequently, an encoding method corresponding to the fifth embodiment will be described. Although the present embodiment differs from the first to fourth embodiments in the target code amount setting method, the configuration of the image processing unit 104 that functions as the imaging device 100 and the encoding unit is the same as that of FIG. It is the same as that shown in 1B. In the present embodiment, weighting is performed for each plane as in the second embodiment, and weighting is performed for each subband as in the third embodiment. Since the basic processing is the same, the description is omitted, and only the parts different from the first to fourth embodiments will be described with reference to FIG.

FIG. 14 (a) shows a case where a Bayer array RAW image is separated into G1, G2, B, and R planes and encoded for each subband obtained by wavelet transform. The case where the code amount is allocated is shown. In this case, the target code amount is equally allocated to each plane and each subband.
When the target code amount for one frame of RAW data is 100%, 25% is allocated to each of the G1, G2, B, and R planes. In the plane, when the target code amount assigned to the plane is 100%, each subband (LL, HL, LH, HH) is assigned a target code amount of 25%. The numerical value in parentheses of each subband indicates the target code amount assigned to each subband when the target code amount for one frame is 100%.

  On the other hand, in FIG. 14B, when generating a sum plane and a difference plane from the G1 / G2 plane and encoding the sum, difference, and B and R planes, weighting is performed according to the plane and subband. Shows the assignment of the target code amount when the target code amount is assigned by performing the above. Assuming that the target code amount of one frame is 100%, 30% is allocated to the sum plane, 20% is allocated to the difference plane, and 25% is allocated to the B / R plane, as in FIG.

  The sum plane corresponds to the green low-frequency component of the RAW data, and the difference plane corresponds to the green high-frequency component. In the image data, the low frequency component is an important component, and if the loss of the low frequency component is large, the image quality deteriorates. Therefore, a large target code amount is allocated to the sum plane. Since the high frequency component is less important than the low frequency component, the difference plane is assigned a smaller target code amount than the sum plane, B plane, and R plane.

  Further, the target code amount of each plane is further assigned for each subband. For the sum plane, the subband target code amount is assigned such that LL: HL: LH: HH = 44: 20: 20: 16. At this time, since the low frequency component is also an important component, a larger amount of target code is allocated to the low frequency sub-band. When the target code amount for one frame is 100%, a plane target code amount of 30% is assigned to the sum plane. Therefore, the ratio of the target code amount of each subband to the target code amount of one frame is 13.2% for LL, 6% for HL, 6% for LH, and 4.8% for HH.

  For the difference plane, the subband target code amount is assigned such that LL: HL: LH: HH = 13: 29: 29: 29. At this time, since the difference plane corresponds to a green high-frequency component, in the difference plane, a higher target code amount is allocated to the high-frequency subband, and the target code amount to the high-frequency subband in the other planes The ratio is larger than the allocation ratio. In the example of FIG. 14B, the allocation ratio of HL, LH, and HH is the same, but HH may be a higher ratio. When the target code amount for one frame is 100%, a 20% plane target code amount is assigned to the difference plane. Therefore, the ratio of the target code amount of each subband to the target code amount of one frame is 2.6% for LL, 5.8% for HL, 5.8% for LH, and 5.8% for HH. Here, the target code amount of the HH subband of the difference plane is larger than the target code amount of the HH subband of the sum plane.

  For the B plane and the R plane, the subband target code amount is assigned so that LL: HL: LH: HH = 25: 25: 25: 25. When the target code amount for one frame is 100%, a plane target code amount of 25% is allocated to the B / R plane. Therefore, the ratio of the target code amount of each subband to the target code amount of one frame is all 6.25%.

  A target code amount given to one frame at such a ratio is assigned to each subband, and quantization and encoding are performed based on the assigned target code amount as described in the above-described embodiment. In this embodiment, the target code amount is assigned to each plane and each subband at the ratio of the target code amount shown in FIG. However, the image may be analyzed as in the fourth embodiment, and the target code amount may be determined by further weighting the ratio of the target code amount in FIG. 14 according to the analysis result. . In this case, as a result of the image analysis, the plane target code amount may be weighted or the subband target code amount may be weighted, and both the plane target code amount and the subband target code amount are weighted. It may be done.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: imaging device, 101: control unit, 102: operation unit

Claims (18)

An image processing apparatus for encoding Bayer array image data,
Color separation means for separating a plurality of color components constituting the image data into a plurality of plane image data based on the Bayer array;
A sum plane generating means for generating a sum plane plane image data by adding components of the same color among the plurality of plane image data;
Difference plane generating means for subtracting the same color components to generate plane image data of a difference plane;
Coding means for compressing and coding the plain image data, the plain image data of the same color component comprising coding means for compressing and coding the plain image data of the sum plane and the difference plane An image processing apparatus. A setting means for setting a target code amount in the compression encoding for each of the plain image data to be compression-encoded by the encoding means;
The image processing apparatus according to claim 1, wherein the setting unit sets the target code amount for the plane image data of the sum plane and the target code amount for the plane image data of the difference plane to be the same. A setting means for setting a target code amount in the compression encoding for each of the plain image data to be compression-encoded by the encoding means;
The image processing apparatus according to claim 1, wherein the setting unit sets a target code amount for the plane image data of the sum plane larger than a target code amount for the plane image data of the difference plane.   The sum of the target code amount for the plane image data of the sum plane and the target code amount for the plane image data of the difference plane is the target code amount for the plane image data of the sum plane and the target code for the plane image data of the difference plane. The image processing apparatus according to claim 3, wherein the amount is the same as the total when the amount is set to be the same. Further comprising conversion means for hierarchically converting the plain image data for each frequency band;
The encoding means compresses and encodes the plain image data converted by the conversion means,
The image processing apparatus according to claim 2, wherein the setting unit sets a target code amount for each subband generated by the conversion unit.   The encoding means has a target code amount of the subband sum plane and difference plane from which low-frequency components are extracted for the plane image data of the sum plane and the plane image data of the difference plane converted by the conversion means. The ratio of the target code amount of the sub-band sum plane and the difference plane from which the high frequency component is extracted to the ratio is compression-encoded so that the target code amount given to the difference plane is larger. The image processing apparatus according to claim 5.   The setting means sets a target code amount so that a larger target code amount is allocated to a low-frequency subband than a high-frequency subband in the sum plane, and a low-frequency subband in the difference plane. 6. The image processing apparatus according to claim 5, wherein the target code amount is set so that a larger target code amount is allocated to a sub-band higher than the band.   The setting means sets the target code amount of the sum plane to be larger than the target code amount of the difference plane, and sets the target code amount of the HH subband of the difference plane to the target code of the HH subband of the sum plane. The image processing apparatus according to claim 5, wherein the image processing apparatus is set to be larger than the amount. Further comprising analysis means for generating analysis information by analyzing image characteristics of the image data;
9. The setting unit according to claim 2, wherein the setting unit sets a target code amount for plane image data of the sum plane and a target code amount for plane image data of the difference plane using the analysis information. The image processing apparatus according to any one of the above.   The image processing apparatus according to claim 9, wherein the analysis information indicates a variance of pixel values in the plain image data of the color component.   The image processing apparatus according to claim 10, wherein the setting unit increases a target code amount for plane image data of the difference plane as the variance increases.   The image processing apparatus according to claim 9, wherein the analysis information is a variance of pixel values in the plane image data of the sum plane.   The image processing apparatus according to claim 9, wherein the analysis information is a variance of pixel values in the plane image data of the difference plane.   The Bayer array includes a first green component, a second green component, a blue component, and a red component, and the same-color components are the first green component and the second green component. The image processing apparatus according to any one of claims 1 to 13.   The Bayer array includes a first red component, a second red component, a first blue component, a second blue component, and a green component, and the same color component includes the first red component and the first red component. The image processing apparatus according to claim 1, wherein the image processing apparatus includes two red components, and the first blue component and the second blue component. Imaging means for capturing a subject image and generating Bayer array image data;
An image processing apparatus comprising: the image processing apparatus according to claim 1, wherein the image processing apparatus encodes the Bayer array image data. An image processing method for encoding Bayer array image data,
A color separation step of separating a plurality of color components constituting the image data into a plurality of plane image data based on the Bayer array;
Of the plurality of plane image data, a sum plane generation step of adding the same color components to generate the plane image data of the sum plane;
A difference plane generating step for generating plane image data of a difference plane by subtracting the same color components;
An encoding step of compressing and encoding the plain image data, the encoding step of compressing and encoding the plain image data of the sum plane and the difference plane for the plain image data of the same color component An image processing method characterized by the above. A program for causing a computer to execute an image processing method for encoding Bayer array image data,
The image processing method includes:
A color separation step of separating a plurality of color components constituting the image data into a plurality of plane image data based on the Bayer array;
Of the plurality of plane image data, a sum plane generation step of adding the same color components to generate the plane image data of the sum plane;
A difference plane generating step for generating plane image data of a difference plane by subtracting the same color components;
An encoding step of compressing and encoding the plane image data, the encoding step of compressing and encoding the plane image data of the sum plane and the difference plane for the plane image data of the same color component A program characterized by

Images