JP4963690B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus, and more particularly to an encoder and a decoder in HD Photo.

  In recent years, HD Photo has been proposed by Microsoft as a still image file format with higher image quality than JPEG and a simplified circuit configuration and arithmetic processing than JPEG2000.

  An encoder in HD Photo includes a frequency conversion unit that performs a predetermined frequency conversion process (PCT) and a prefilter that executes a predetermined prefilter process for reducing block distortion. The frequency conversion unit performs the frequency conversion process using a pixel block of 4 pixels vertically × 4 pixels horizontally as a processing unit region. The pre-filter performs pre-filter processing before the frequency conversion processing is executed, with a region overlapping with a plurality of processing unit regions by the frequency conversion unit as a processing unit region. However, whether or not to execute the pre-filter process can be arbitrarily selected by setting an overlap coefficient.

  The decoder in HD Photo includes a frequency inverse transform unit that performs a frequency inverse transform process corresponding to the frequency transform process, and a post filter that performs a post filter process corresponding to the prefilter process. Has been. The frequency inverse transform unit performs frequency inverse transform processing using a vertical 4 pixel × horizontal 4 pixel pixel block as a processing unit region. The post-filter performs post-filter processing after frequency inverse transform processing is performed, with a region that overlaps a plurality of processing unit regions by the frequency inverse transform unit as a processing unit region. However, in the same manner as described above, whether or not to execute the post filter processing can be arbitrarily selected by setting an overlap coefficient.

  The details of HD Photo are disclosed in Non-Patent Document 1, for example. Details of JPEG XR related to HD Photo are disclosed in Non-Patent Document 2 below, for example.

JP 2005-333251 A "HD Photo -Photographic Still Image File Format", [online], November 7, 2006, Microsoft Corporation, [October 10, 2007 search], Internet <URL: http://www.microsoft.com/whdc /xps/hdphotodpk.mspx> "Coding of Still Pictures -JBIG JPEG", [online], 19 December 2007, ISO / IEC JTC 1 / SC 29 / WG1 N 4392, [March 4, 2008 search], Internet <URL: http: // www .itscj.ipsj.or.jp / sc29 / open / 29view / 29n9026t.doc>

  The encoder in HD Photo supports RAW data acquired by a Bayer array image sensor, but supports RAW data acquired by a honeycomb array image sensor such as Super CCD Honeycomb (registered trademark). Not done.

  The present invention has been made in view of such circumstances, and an object thereof is to obtain an image processing apparatus capable of handling RAW data acquired by an image pickup device having a honeycomb array while minimizing circuit changes. .

  The image processing apparatus according to the first aspect of the present invention provides a color for converting a signal in the first color space acquired by the honeycomb array of imaging elements into a signal in the second color space based on a predetermined arithmetic expression. A conversion unit, wherein the color conversion unit defines pixels including cells of a plurality of colors in the honeycomb arrangement, performs a color conversion process on a pixel-by-pixel basis, and a pixel to be processed by the color conversion unit includes a pixel The arrangement of each color cell includes a first pixel that is a first arrangement pattern and a second pixel that is a second arrangement pattern. In the arithmetic expression, the signal value of each cell is Can be set as a parameter, and when the first pixel is processed, the color conversion unit sets a first parameter, and when the second pixel is processed, The position of the signal value of each color cell is the first pixel. It equally becomes as related, it is characterized in setting the second parameter.

  According to the image processing apparatus according to the first aspect, the difference in the arrangement pattern of the cells of each color in the pixel with respect to both the first and second pixels by a simple method of changing the parameter setting in the arithmetic expression. Therefore, it is possible to execute an appropriate color conversion process considering the above.

  An image processing apparatus according to a second aspect of the present invention includes a color conversion unit that converts a signal of a first color space acquired by an image pickup device having a honeycomb array into a signal of a second color space; A rearrangement processing unit that rearranges the signals of the first color space before the color space signal is input to the color conversion unit, and the color conversion unit includes a plurality of color cells in the honeycomb arrangement. The color conversion process is executed in units of pixels by defining the pixels to be included, and the arrangement of the cells of each color in the pixel is the first arrangement pattern for the pixel that is the processing target of the color conversion unit. If the pixel and the second pixel that is the second arrangement pattern are included, the rearrangement processing unit executes the rearrangement process when the processing target of the color conversion unit is the first pixel. If the processing target of the color conversion unit is the second pixel, As the position of the signal value of each color cell in the inner becomes equal to that for the first pixel, it is characterized in performing the reordering process.

  According to the image processing apparatus according to the second aspect, the cells of each color in the pixel are related to both the first and second pixels by a simple technique of rearranging the signal value positions of the cells of each color in the pixel. Therefore, it is possible to execute an appropriate color conversion process in consideration of the difference in the arrangement pattern.

  An image processing apparatus according to a third aspect of the present invention includes a color conversion unit that converts a signal of a first color space acquired by an image pickup device having a honeycomb array into a signal of a second color space, and the color conversion unit The color conversion unit defines a pixel including cells of a plurality of colors in the honeycomb arrangement, executes color conversion processing on a pixel basis, and is processed by the color conversion unit. In the pixel, the arrangement of the cells of each color in the pixel includes a first pixel that is a first arrangement pattern and a second pixel that is a second arrangement pattern. A first conversion processing unit that performs color conversion processing based on the first arithmetic expression, and a position where the signal value position of each color cell is equivalently equal to that in the first arithmetic expression. A second conversion processing unit that performs color conversion processing based on the arithmetic expression of 2; And the selection unit selects the first conversion processing unit when the processing target of the color conversion unit is the first pixel, and the processing target of the color conversion unit is the second pixel. In the case of a pixel, the second conversion processing unit is selected.

  According to the image processing apparatus according to the third aspect, the first conversion processing unit and the second conversion processing unit are provided in the color conversion unit, and a first method is selected by selecting one according to the processing target. For both the second pixel and the second pixel, an appropriate color conversion process can be executed in consideration of the difference in the arrangement pattern of the cells of each color in the pixel.

  An image processing apparatus according to a fourth aspect of the present invention includes a color conversion unit that converts a signal of a first color space acquired by an image pickup device having a honeycomb array into a signal of a second color space, and the color conversion The section defines pixels including cells of a plurality of colors in the honeycomb arrangement, and performs color conversion processing in units of pixels, and the color conversion section has a single arrangement pattern in which the cells of each color in the pixel are arranged. Thus, the pixel is defined as described above.

  According to the image processing apparatus according to the fourth aspect, by defining the pixels so that the arrangement of the cells of each color in the pixel becomes a single arrangement pattern, the arrangement pattern of the cells of each color is changed in all the pixels. It will be the same. Therefore, it is possible to execute appropriate color conversion processing for all pixels without performing processing such as parameter setting change or rearrangement of signal value positions.

  The image processing apparatus according to the fifth aspect of the present invention relates to a shortage cell particularly for a pixel whose number of cells in one pixel is insufficient at the peripheral portion of the pixel plane in the image processing apparatus according to the fourth aspect. A complement processing unit that complements the signal value is further provided, and the complement processing unit supplements the signal value related to the insufficient cell by calculation using the signal value related to the cell located in the vicinity of the insufficient cell. It is what.

  With the image processing apparatus according to the fifth aspect, it is possible to increase the compression efficiency by performing high-precision complementation by calculation.

  The image processing apparatus according to the sixth aspect of the present invention relates to a shortage cell, particularly for the pixel whose number of cells in one pixel is insufficient at the peripheral portion of the pixel plane in the image processing apparatus according to the fourth aspect. A complement processing unit for complementing signal values is further provided, wherein the complement processing unit supplements signal values related to the insufficient cells using a predetermined value.

  According to the image processing apparatus of the sixth aspect, it is possible to reduce the processing load by performing complementation using a predetermined value without performing calculation.

  An image processing apparatus according to a seventh aspect of the present invention is the image processing apparatus according to any one of the first to sixth aspects, and in particular, the color conversion unit receives information relating to the definition of the pixel, A signal in the second color space is output, in which information on the position of each color cell is included in the header portion.

  According to the image processing apparatus of the seventh aspect, the information related to the definition of the pixel and the information related to the position of the cell of each color in the pixel are included in the header portion of the signal in the second color space. Therefore, the color inverse conversion unit that inversely converts the signal in the second color space into the signal in the first color space can appropriately reproduce the original image by referring to the information.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus which can respond | correspond to the RAW data acquired by the image pick-up element of a honeycomb arrangement | sequence can be obtained, suppressing the change of a circuit to the minimum.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, the element which attached | subjected the same code | symbol in different drawing shall show the same or corresponding element. Hereinafter, an example in which the image processing apparatus according to the present invention is applied to an encoder and a decoder in HD Photo will be described.

<First Embodiment>
FIG. 1 is a block diagram showing a configuration of an image processing apparatus 1 according to the first embodiment of the present invention. The image processing apparatus 1 includes an encoder 2 and a decoder 3. The encoder 2 includes a color conversion unit 21, a prefilter 22, a frequency conversion unit 23, a quantization unit 24, a prediction unit 25, and an encoding unit 26. The decoder 3 includes a color inverse transform unit 31, a post filter 32, a frequency inverse transform unit 33, an inverse quantization unit 34, an inverse prediction unit 35, and a decoding unit 36.

  The encoding unit 26 and the decoding unit 36 are connected to the memory 4. The color conversion unit 21 is connected to the buffer memory 11, and the buffer memory 11 is connected to the image sensor 10. However, the buffer memory 11 and the memory 4 may be combined. The image sensor 10 is an image sensor in which a color filter in a honeycomb arrangement, such as a super CCD honeycomb (registered trademark), is arranged. A RAW image signal S1 acquired by the image sensor 10 is input to the buffer memory 11 and temporarily stored.

  FIG. 2 is a diagram illustrating a RAW image acquired by the imaging element 10 having the honeycomb arrangement. In FIG. 2, only the upper left corner of the RAW image is extracted and shown. In the example shown in FIG. 2, in even lines, red cells in which red (R) color filters are arranged and blue cells in which blue (B) color filters are arranged are alternately arranged. . In the odd lines, green cells in which green (G) color filters are arranged are arranged. The odd lines are arranged with a shift of ½ the cell pitch with respect to the even lines.

  In the image processing apparatus 1 according to the present embodiment, each green cell (G) in the honeycomb array is divided into a green cell (Gr) adjacent to the red cell and a green cell (Gb) adjacent to the blue cell in the Bayer array. Correspond to one of the.

  FIG. 3 is a view showing a RAW image similar to FIG. The green cell (G) sandwiched between the upper left red cell (R) and the lower right red cell (R) is associated with the green cell (Gr) in the Bayer array, and the upper left blue cell (B) The green cell (G) sandwiched between the lower right blue cell (B) is associated with the green cell (Gb) in the Bayer array. Thereby, a RAW image expressed in the RGrGbB color space is obtained. Contrary to the above, the green cell (G) sandwiched between the upper right red cell (R) and the lower left red cell (R) is associated with the green cell (Gr), and the upper right blue cell (B) The green cell (G) sandwiched between the lower left blue cell (B) may be associated with the green cell (Gb).

  Referring to FIG. 1, the signal S <b> 1 read from the buffer memory 11 is input to the color conversion unit 21. The color converter 21 converts the signal S1 in the RGrGbB color space into a pixel signal S2 in the YUVK color space and outputs the pixel signal S2. Details of the color conversion processing in the color conversion unit 21 will be described later.

  The pixel signal S <b> 2 is input from the color conversion unit 21 to the prefilter 22. The prefilter 22 selectively executes a predetermined prefilter process on the pixel signal S2 and outputs a pixel signal S3.

  The pixel signal S <b> 3 is input from the prefilter 22 to the frequency conversion unit 23. The frequency conversion unit 23 performs a predetermined frequency conversion process (PCT: HD Photo Core Transform) on the pixel signal S3, and outputs frequency data S4 after the frequency conversion process. In HD Photo, the frequency data S4 includes a high-pass component, a low-pass component, and a DC component.

  The frequency data S4 is input from the frequency conversion unit 23 to the quantization unit 24. The quantization unit 24 quantizes the frequency data S4 and outputs the quantized frequency data S5.

  The frequency data S5 is input from the quantization unit 24 to the prediction unit 25. The prediction unit 25 obtains a difference value between the value of the frequency data S5 input from the quantization unit 24 and the value (prediction value) of specific frequency data processed in the past, and outputs the difference value as frequency difference data S6.

  The encoding unit 26 receives the frequency difference data S6 from the prediction unit 25. The encoding unit 26 performs entropy encoding on the frequency difference data S6 and outputs encoded data S7. The encoded data S7 output from the encoding unit 26 is stored in the memory 4. Strictly speaking, in HD Photo, entropy encoding is performed only on the Normal Bit corresponding to the upper bits in the frequency difference data S6, and the Entropy encoding is performed on the Flex Bit corresponding to the lower bits. It is stored in the memory 4 without being executed.

  The encoded data S <b> 7 is input from the memory 4 to the decoding unit 36. The decoding unit 36 performs entropy decoding on the encoded data S7, and outputs frequency difference data S8 corresponding to the frequency difference data S6.

  The inverse prediction unit 35 receives the frequency difference data S8 from the decoding unit 36. The inverse prediction unit 35 outputs the frequency data S9 corresponding to the frequency data S5 by adding the predicted value to the frequency difference data S8.

  The inverse quantization unit 34 receives the frequency data S9 from the inverse prediction unit 35. The inverse quantization unit 34 inversely quantizes the frequency data S9 and outputs frequency data S10 corresponding to the frequency data S4.

  The frequency data S <b> 10 is input from the inverse quantization unit 34 to the frequency inverse transform unit 33. The frequency inverse conversion unit 33 performs frequency inverse conversion processing corresponding to the frequency conversion processing on the frequency data S10, and outputs a pixel signal S11 corresponding to the pixel signal S3.

  The pixel signal S <b> 11 is input from the frequency inverse conversion unit 33 to the post filter 32. The post filter 32 selectively executes post filter processing corresponding to the pre-filter processing on the pixel signal S11, and outputs a pixel signal S12 corresponding to the pixel signal S2.

  The pixel signal S <b> 12 is input from the post filter 32 to the color inverse conversion unit 31. The color reverse conversion unit 31 converts the pixel signal S12 in the YUVK color space into the pixel signal S13 in the RGrGbB color space, and outputs it to an external display device or the like.

  Hereinafter, the color conversion processing in the color conversion unit 21 will be described.

  First, the color conversion unit 21 defines a pixel that is a unit of color conversion processing in the RAW image shown in FIG. One pixel includes one red cell (R), one blue cell (B), one green cell (Gr), and one green cell (Gb).

  FIG. 4 is a diagram illustrating a first example of pixel definition. Each pixel includes a red cell (R) and a blue cell (B) belonging to the same cell row, and a green cell (Gr) and a green cell (Gb) belonging to the cell row below the pixel. As a result, in the pixels G00 to G02 and G20 to G22 belonging to the even pixel rows Y0 and Y2, the upper left is the red cell (R), the upper right is the blue cell (B), the lower left is the green cell (Gr), and the lower right is In the pixels G10 to G12 and G30 to G32 belonging to the odd pixel rows Y1 and Y3, the upper left is the blue cell (B), the upper right is the red cell (R), and the lower left is the green cell (Gb). The lower right is a green cell (Gr).

  The color conversion unit 21 performs color conversion processing in units of pixels in order from the upper left pixel. That is, color conversion processing is executed in the order of pixels G00 → G01 → G02 →... → G10 → G11 → G12 →... → G20 → G21 → G22 →... → G30 → G31 → G32 →. To do.

  FIG. 5 is a diagram schematically showing the configuration of the color conversion unit 21. The color conversion unit 21 includes four registers F1 to F4 and a calculation unit 13 connected to these registers F1 to F4. Of the four registers, the upper left is the register F1, the upper right is the register F2, the lower left is the register F3, and the lower right is the register F4.

  A signal value related to the upper left cell in the pixel is input to the upper left register F1, a signal value related to the upper right cell in the pixel is input to the upper right register F2, and a signal value related to the upper left cell in the pixel is input to the lower left register F3. A signal value related to the lower left cell is input, and a signal value related to the lower right cell in the pixel is input to the lower right register F4.

  Therefore, when processing the pixels G00 to G02 and G20 to G22 belonging to the even pixel rows Y0 and Y2, the signal value of the red cell (R) is input to the register F1, and the blue cell (B ), The signal value of the green cell (Gr) is input to the register F3, and the signal value of the green cell (Gb) is input to the register F4. When the pixels G10 to G12 and G30 to G32 belonging to the odd pixel rows Y1 and Y3 are processed, the signal value of the blue cell (B) is input to the register F1, and the red cell (R) is input to the register F2. ), The signal value of the green cell (Gb) is input to the register F3, and the signal value of the green cell (Gr) is input to the register F4.

  FIG. 6 is a diagram illustrating an arithmetic expression for the arithmetic unit 13 to perform color conversion processing by arithmetic operation. As shown in FIG. 6, the arithmetic expression is described using parameters P1 to P4. The parameters P1 to P4 are set in the registers F1 to F4 shown in FIG. The color conversion unit 21 calculates each pixel value of the K signal, the V signal, the U signal, and the Y signal based on the arithmetic expression shown in FIG. In the conversion formula shown in FIG. 6, “floor (x)” means the largest integer not more than x, and “ceiling (x)” means the smallest integer not less than x. “Α” means 1/2 of the number of gradations of the pixel value. For example, when the pixel value is represented by 8 bits, since there are 256 gradations, “α” is 128.

  FIG. 7 is a diagram illustrating an example of setting the parameters P1 to P4 in the registers F1 to F4. When processing the pixels G00 to G02 and G20 to G22 belonging to the even pixel rows Y0 and Y2, the color conversion unit 21 stores the registers F1, F2, F3, and F4 in the registers F1, F2, F3, and F4 as shown in FIG. , Parameters P4, P3, P1 and P2 are set, respectively. Further, when the pixels G10 to G12 and G30 to G32 belonging to the odd pixel rows Y1 and Y3 are processed, the color converter 21 registers F1, F2, F3, as shown in FIG. Parameters P3, P4, P2 and P1 are set in F4, respectively.

  FIG. 8 is a diagram showing a conversion formula from the RGrGbB color space to the YUVK color space in HD Photo. By setting the parameters P1 to P4 in the registers F1 to F4 as shown in FIGS. 7A and 7B, all the pixels G00 to G02, G10 to G12, G20 to G22, and G30 to G32 are related. The conversion formula shown in FIG. 8 is realized.

  FIG. 9 is a diagram illustrating a second example of pixel definition. Each pixel includes a red cell (R) and a blue cell (B) belonging to the same cell column, and a green cell (Gr) and a green cell (Gb) belonging to the adjacent cell column. As a result, in the pixels G00, G10, G02, G12, G04, and G14 belonging to the even pixel rows X0, X2, and X4, the upper left is the red cell (R), the upper right is the green cell (Gr), and the lower left is the blue cell ( B), the lower right is a green cell (Gb), and in the pixels G01, G11, G03, G13, G05, and G15 belonging to the odd pixel columns X1, X3, and X5, the upper left is a blue cell (B), and the upper right is green. The cell (Gb), the lower left is a red cell (R), and the lower right is a green cell (Gr).

  The color conversion unit 21 performs color conversion processing in units of pixels in order from the upper left pixel. That is, the color conversion processing is executed in the order of pixels G00 → G01 → G02 → G03 → G04 → G05 →... → G10 → G11 → G12 → G13 → G14 → G15 →.

  FIG. 10 is a diagram illustrating an example of setting the parameters P1 to P4 in the registers F1 to F4. When processing the pixels G00, G10, G02, G12, G04, and G14 belonging to the even-numbered pixel columns X0, X2, and X4, as shown in FIG. Parameters P4, P1, P3, and P2 are set in F2, F3, and F4, respectively. Further, when processing the pixels G01, G11, G03, G13, G05, and G15 belonging to the odd pixel columns X1, X3, and X5, the color conversion unit 21 registers the registers as shown in FIG. Parameters P3, P2, P4, and P1 are set in F1, F2, F3, and F4, respectively. By defining the parameters P1 to P4 in the registers F1 to F4 as shown in (A) and (B) of FIG. 10, the conversion formulas shown in FIG. 8 are obtained for all the pixels G00 to G05 and G10 to G15. Is realized.

  In the present embodiment, the color conversion unit 21 defines a pixel including cells of a plurality of colors in the honeycomb arrangement in the RAW image, and executes a color conversion process on a pixel basis. Here, in the pixel to be processed by the color conversion unit 21, the arrangement of the cells of each color in the pixel is the first pixel (for example, the pixel G00 in FIG. 4) which is the first arrangement pattern, and the first A second pixel that is a second arrangement pattern different from the arrangement pattern (for example, pixel G10 in FIG. 4) is included. Further, as shown in FIG. 6, in the arithmetic expression, the signal value of each cell can be set as parameters P1 to P4. When the first pixel is processed, the color conversion unit 21 sets a first parameter (for example, (A) in FIG. 7). On the other hand, when the second pixel is processed, a second parameter (for example, (B) in FIG. 7) is set such that the position of the signal value of the cell of each color in the arithmetic expression is equal to that of the first pixel. ) Is set.

  Therefore, according to the present embodiment, the difference in the arrangement pattern of the cells of the respective colors in the pixel with respect to both the first and second pixels by a simple method of changing the setting of the parameters P1 to P4 in the arithmetic expression. Therefore, it is possible to execute an appropriate color conversion process considering the above. As a result, it is possible to avoid the possibility that the encoding efficiency is deteriorated.

<Second Embodiment>
FIG. 11 is a block diagram showing a part of the configuration of the image processing apparatus 1 according to the second embodiment of the present invention. A rearrangement processing unit 15 is connected between the buffer memory 11 and the color conversion unit 21. The rearrangement processing unit 15 selectively rearranges the signal S1 read from the buffer memory 11, and inputs the signal S1 to the color conversion unit 21 as the signal S1a.

  FIG. 12 is a diagram illustrating a first example of the registers F1 to F4 included in the color conversion unit 21. As illustrated in FIG. In this example, the signal value of the red cell (R) is input to the register F1, the signal value of the blue cell (B) is input to the register F2, and the signal value of the green cell (Gr) is input to the register F3. The signal value of the green cell (Gb) is input to the register F4. The calculation unit 13 (not shown in FIG. 12) included in the color conversion unit 21 executes color conversion processing from the RGrGbB color space to the YUVK color space based on the calculation formula shown in FIG.

  In the example shown in FIG. 4, in the pixels G00 to G02 and G20 to G22 belonging to the even pixel rows Y0 and Y2, the upper left is the red cell (R), the upper right is the blue cell (B), and the lower left is the green cell (Gr ), The lower right is a green cell (Gb). Accordingly, since the arrangement pattern of each color in the pixel matches the arrangement pattern of each color in the registers F1 to F4 shown in FIG. 12, the rearrangement processing unit 15 does not perform the rearrangement process, and the cell of each color. Are input to the corresponding registers F1 to F4.

  On the other hand, in the pixels G10 to G12 and G30 to G32 belonging to the odd pixel rows Y1 and Y3, the upper left is a blue cell (B), the upper right is a red cell (R), the lower left is a green cell (Gb), and the lower right is green. This becomes a cell (Gr). Accordingly, the arrangement pattern of each color in the registers F1 to F4 shown in FIG. 12 is reversed to the left and right with respect to the arrangement pattern of each color in the pixel. For this reason, the rearrangement processing unit 15 horizontally inverts the signal value of each color cell in the pixel, and then inputs the signal value of each color cell to the corresponding registers F1 to F4.

  FIG. 13 is a diagram illustrating a second example of the registers F1 to F4 included in the color conversion unit 21. As illustrated in FIG. In this example, the signal value of the red cell (R) is input to the register F1, the signal value of the green cell (Gr) is input to the register F2, and the signal value of the blue cell (B) is input to the register F3. The signal value of the green cell (Gb) is input to the register F4. The calculation unit 13 (not shown in FIG. 12) included in the color conversion unit 21 executes color conversion processing from the RGrGbB color space to the YUVK color space based on the calculation formula shown in FIG.

  In the example shown in FIG. 9, in the pixels G00, G10, G02, G12, G04, and G14 belonging to the even pixel rows X0, X2, and X4, the upper left is the red cell (R), the upper right is the green cell (Gr), The lower left is a blue cell (B) and the lower right is a green cell (Gb). Therefore, since the arrangement pattern of each color in the pixel matches the arrangement pattern of each color in the registers F1 to F4 shown in FIG. 13, the rearrangement processing unit 15 does not execute the rearrangement process, and the cell of each color. Are input to the corresponding registers F1 to F4.

  On the other hand, in the pixels G01, G11, G03, G13, G05, and G15 belonging to the odd pixel columns X1, X3, and X5, the upper left is a blue cell (B), the upper right is a green cell (Gb), and the lower left is a red cell (R ), The lower right is a green cell (Gr). Therefore, the arrangement pattern of each color in the registers F1 to F4 shown in FIG. 13 is reversed upside down with respect to the arrangement pattern of each color in the pixel. For this reason, the rearrangement processing unit 15 vertically inverts the signal value of each color cell in the pixel, and then inputs the signal value of each color cell to the corresponding registers F1 to F4.

  FIG. 14 is a diagram illustrating a modification of the second embodiment. As shown in FIG. 14, the rearrangement processing unit 15 may be connected between the image sensor 10 and the buffer memory 11.

  The image processing apparatus 1 according to the present embodiment includes a rearrangement processing unit 15 that is arranged before the color conversion unit 21. In addition, the color conversion unit 21 defines pixels including cells of a plurality of colors in the honeycomb arrangement in the RAW image, and executes the color conversion process in units of pixels. Here, in the pixel to be processed by the color conversion unit 21, the arrangement of the cells of each color in the pixel is the first pixel (for example, the pixel G00 in FIG. 4) which is the first arrangement pattern, and the first A second pixel that is a second arrangement pattern different from the arrangement pattern (for example, pixel G10 in FIG. 4) is included. Then, when the processing target of the color conversion unit 21 is the first pixel, the rearrangement processing unit 15 does not execute the rearrangement processing and the processing target of the color conversion unit 21 is the second pixel. The rearrangement process is executed so that the position of the signal value of the cell of each color in the pixel is equal to that of the first pixel.

  Therefore, according to the present embodiment, the arrangement pattern of the cells of each color in the pixel with respect to both the first and second pixels can be obtained by a simple technique of rearranging the signal value positions of the cells of each color in the pixel. Appropriate color conversion processing can be executed in consideration of these differences. As a result, it is possible to avoid the possibility that the encoding efficiency is deteriorated.

<Third Embodiment>
FIG. 15 is a block diagram showing a part of the configuration of the image processing apparatus 1 according to the third embodiment of the present invention. A selection unit 40 is connected in front of the color conversion unit 21. The color conversion unit 21 includes conversion processing units 41A and 41B. The selection unit 40 selects one of the conversion processing units 41A and 41B and inputs the signal S1.

  FIG. 16 is a diagram schematically illustrating a first example of the configuration of the conversion processing unit 41A. As illustrated in FIG. 16A, the conversion processing unit 41A includes four registers F1 to F4 and an arithmetic unit 13A connected to these registers F1 to F4. FIG. 16B shows an arithmetic expression for the arithmetic unit 13A to execute color conversion processing by arithmetic operation. For example, the value of the K signal is obtained by subtracting the value Q4 of the register F4 from the value Q3 of the register F3, and the value of the V signal is obtained by subtracting the value Q1 of the register F1 from the value Q2 of the register F2.

  FIG. 17 is a diagram schematically illustrating a first example of the configuration of the conversion processing unit 41B. As shown in FIG. 17A, the conversion processing unit 41B includes four registers F1 to F4 and an arithmetic unit 13B connected to these registers F1 to F4. FIG. 17B shows an arithmetic expression for the arithmetic unit 13B to perform color conversion processing by arithmetic operation. For example, the value of the K signal is obtained by subtracting the value Q3 of the register F3 from the value Q4 of the register F4, and the value of the V signal is obtained by subtracting the value Q2 of the register F2 from the value Q1 of the register F1.

  In the example illustrated in FIG. 4, when processing the pixels G00 to G02 and G20 to G22 belonging to the even pixel rows Y0 and Y2, the selection unit 40 selects the conversion processing unit 41A. As a result, the value Q1 of the register F1 becomes the signal value of the red cell (R), the value Q2 of the register F2 becomes the signal value of the blue cell (B), and the value Q3 of the register F3 becomes the signal value of the green cell (Gr). The value Q4 of the register F4 becomes the signal value of the green cell (Gb). As a result, the arithmetic expression shown in FIG. 16B is equivalently equivalent to the arithmetic expression shown in FIG.

  On the other hand, when processing the pixels G10 to G12 and G30 to G32 belonging to the odd pixel rows Y1 and Y3, the selection unit 40 selects the conversion processing unit 41B. Thus, the value Q1 of the register F1 becomes the signal value of the blue cell (B), the value Q2 of the register F2 becomes the signal value of the red cell (R), and the value Q3 of the register F3 becomes the signal value of the green cell (Gb). The value Q4 of the register F4 becomes the signal value of the green cell (Gr). As a result, the arithmetic expression shown in FIG. 17B is equivalently equivalent to the arithmetic expression shown in FIG.

  FIG. 18 is a diagram schematically illustrating a second example of the configuration of the conversion processing unit 41A. As illustrated in FIG. 18A, the conversion processing unit 41A includes four registers F1 to F4 and an arithmetic unit 13A connected to these registers F1 to F4. FIG. 18B shows an arithmetic expression for the arithmetic unit 13A to execute color conversion processing by arithmetic operation. For example, the value of the K signal is obtained by subtracting the value Q4 of the register F4 from the value Q2 of the register F2, and the value of the V signal is obtained by subtracting the value Q1 of the register F1 from the value Q3 of the register F3.

  FIG. 19 is a diagram schematically illustrating a second example of the configuration of the conversion processing unit 41B. As shown in FIG. 19A, the conversion processing unit 41B includes four registers F1 to F4 and a calculation unit 13B connected to these registers F1 to F4. FIG. 19B shows an arithmetic expression for the arithmetic unit 13B to perform color conversion processing by arithmetic operation. For example, the value of the K signal is obtained by subtracting the value Q2 of the register F2 from the value Q4 of the register F4, and the value of the V signal is obtained by subtracting the value Q3 of the register F3 from the value Q1 of the register F1.

  In the example shown in FIG. 9, when processing the pixels G00, G10, G02, G12, G04, and G14 belonging to the even pixel rows X0, X2, and X4, the selection unit 40 selects the conversion processing unit 41A. As a result, the value Q1 of the register F1 becomes the signal value of the red cell (R), the value Q2 of the register F2 becomes the signal value of the green cell (Gr), and the value Q3 of the register F3 becomes the signal value of the blue cell (B). The value Q4 of the register F4 becomes the signal value of the green cell (Gb). As a result, the arithmetic expression shown in FIG. 18B is equivalently equivalent to the arithmetic expression shown in FIG.

  On the other hand, when processing the pixels G01, G11, G03, G13, G05, and G15 belonging to the odd pixel rows X1, X3, and X5, the selection unit 40 selects the conversion processing unit 41B. Thus, the value Q1 of the register F1 becomes the signal value of the blue cell (B), the value Q2 of the register F2 becomes the signal value of the green cell (Gb), and the value Q3 of the register F3 becomes the signal value of the red cell (R). The value Q4 of the register F4 becomes the signal value of the green cell (Gr). As a result, the arithmetic expression shown in FIG. 19B is equivalently equivalent to the arithmetic expression shown in FIG.

  The image processing apparatus 1 according to the present embodiment includes a selection unit 40 arranged in the preceding stage of the color conversion unit 21. In addition, the color conversion unit 21 defines pixels including cells of a plurality of colors in the honeycomb arrangement in the RAW image, and executes the color conversion process in units of pixels. Here, in the pixel to be processed by the color conversion unit 21, the arrangement of the cells of each color in the pixel is the first pixel (for example, the pixel G00 in FIG. 4) which is the first arrangement pattern, and the first A second pixel that is a second arrangement pattern different from the arrangement pattern (for example, pixel G10 in FIG. 4) is included. Further, the color conversion unit 21 includes a conversion processing unit 41A that executes color conversion processing based on a first arithmetic expression (for example, FIG. 16B), and a second arithmetic expression (for example, FIG. 17B). ) Based on the color conversion processing unit 41B. Here, the second arithmetic expression is an arithmetic expression such that the position of the signal value of each color cell is equivalently equal to that in the first arithmetic expression. The selection unit 40 selects the conversion processing unit 41A when the processing target of the color conversion unit 21 is the first pixel, and when the processing target of the color conversion unit 21 is the second pixel. The conversion processor 41B is selected.

  Therefore, according to the present embodiment, the first and second pixels are obtained by a simple method of providing the conversion processing unit 41A and the conversion processing unit 41B in the color conversion unit 21 and selecting one according to the processing target. In both cases, it is possible to execute appropriate color conversion processing in consideration of the difference in the arrangement pattern of the cells of each color in the pixel. As a result, it is possible to avoid the possibility that the encoding efficiency is deteriorated.

  Note that the conversion processing unit 41A and the conversion processing unit 41B may be configured to share some elements (for example, the calculation units 13A and 13B).

<Fourth embodiment>
FIG. 20 is a diagram illustrating a first example of pixel definition. Each pixel includes a blue cell (B) belonging to a certain cell row, a green cell (Gr) and a green cell (Gb) belonging to the cell row below, and a red cell (R) belonging to the cell row below the cell. It is included. As a result, in each pixel, the top is a blue cell (B), the left is a green cell (Gr), the right is a green cell (Gb), and the bottom is a red cell (R).

  FIG. 21 is a block diagram showing a part of the configuration of the image processing apparatus 1 according to the fourth embodiment of the present invention. A complement processing unit 50 is connected between the buffer memory 11 and the color conversion unit 21. The complement processing unit 50 complements the signal value related to the insufficient cell for pixels where the number of cells in one pixel is insufficient at the peripheral portion of the pixel plane, and inputs the complemented signal S1b to the color conversion unit 21. .

  As a first example of the complement processing, the complement processing unit 50 supplements the signal value related to the insufficient cell by calculation using the signal value related to the cell located in the vicinity of the insufficient cell.

  Referring to FIG. 20, for example, regarding the pixel G20, the signal value relating to the green cell (Gr) is insufficient. Therefore, the complement processing unit 50 supplements the signal value related to the green cell (Gb) in the pixel G20 as the signal value related to the green cell (Gr) in the pixel G20.

  For example, for the pixel G01, the signal value for the blue cell (B), the signal value for the green cell (Gr), and the signal value for the green cell (Gb) are insufficient. Therefore, the complement processing unit 50 calculates an average value of the signal value related to the blue cell (B) in the pixel G10 and the signal value related to the blue cell (B) in the pixel G11, and the value is calculated as the value in the pixel G01. It complements as a signal value regarding a blue cell (B). The complement processing unit 50 complements the signal value related to the green cell (Gb) in the pixel G10 as the signal value related to the green cell (Gr) in the pixel G01, and the signal value related to the green cell (Gr) in the pixel G11. Is supplemented as a signal value relating to the green cell (Gb) in the pixel G01.

  For example, for the pixel G00, the signal value for the blue cell (B), the signal value for the green cell (Gr), and the signal value for the green cell (Gb) are insufficient. Therefore, the complement processing unit 50 supplements the signal value related to the blue cell (B) in the pixel G10 as the signal value related to the blue cell (B) in the pixel G00. In addition, the complement processing unit 50 supplements the signal value related to the green cell (Gr) in the pixel G10 as each signal value related to the green cell (Gr) and the green cell (Gb) in the pixel G00.

  As a second example of the complement processing, the complement processing unit 50 supplements signal values related to the insufficient cells using a predetermined value.

  For example, when the signal value is represented by 256 gradations, the intermediate 128th gradation value is complemented as the signal values of all the insufficient cells.

  Further, for example, the signal value of the red cell (R) in each pixel is complemented as the signal value of all the insufficient cells in that pixel.

  FIG. 22 is a diagram illustrating a second example of pixel definition. Each pixel includes a red cell (R) and a blue cell (B) belonging to a certain cell row, and a green cell (Gr) and a green cell (Gb) belonging to a cell row below the pixel. As a result, in each pixel, the upper left is a red cell (R), the upper right is a blue cell (B), the lower left is a green cell (Gr), and the lower right is a green cell (Gb). The shortage cell compensation is the same as described above.

  FIG. 23 is a diagram illustrating a third example of pixel definition. Each pixel has a red cell (R) belonging to a certain cell row, a green cell (Gr) belonging to the cell row below it, a blue cell (B) belonging to the cell row below it, and a cell row below it And a green cell (Gb) belonging to. As a result, in each pixel, the upper left is a red cell (R), the upper right is a green cell (Gr), the lower left is a blue cell (B), and the lower right is a green cell (Gb). The shortage cell compensation is the same as described above.

  The color conversion unit 21 according to the present embodiment defines the pixels in the RAW image so that the arrangement of the cells of each color in the pixel becomes a single arrangement pattern. Accordingly, the arrangement pattern of the cells of each color is the same for all the pixels in the RAW image. Therefore, parameter setting change as in the first embodiment, rearrangement processing as in the second embodiment, and switching processing of the conversion processing units 41A and 41B as in the third embodiment are performed. Without performing, an appropriate color conversion process considering the difference in the arrangement pattern of the cells of each color in the pixel can be executed for all the pixels. As a result, it is possible to avoid the possibility that the encoding efficiency is deteriorated.

  Further, according to the first example of the complementing process, the compression efficiency can be increased by performing the highly accurate complementing by calculation.

  Moreover, according to the 2nd example of a complementation process, a process load can be reduced by performing complementation using a predetermined value, without calculating.

<About passing information to the decoder 3>
FIG. 24 is a diagram illustrating the pixel signal S2 output from the color conversion unit 21. The pixel signal S2 has a header portion H and a payload portion P. In the header portion H, information relating to the definition of pixels in the RAW image by the color conversion portion 21, that is, information regarding what pattern (cutting-out method) the pixels are defined in the RAW image is described as data D1. Has been. Further, in the header portion H, information on the position of each color cell in the pixel, that is, information on which signal value in the pixel corresponds to which color is described as data D2. Referring to FIG. 1, data D1 and D2 added to the header portion H of the signal S2 by the color conversion unit 21 are delivered to the color reverse conversion unit 31 along with the signals S3 to S12.

  FIG. 25 is a diagram illustrating an arithmetic expression for the color reverse conversion unit 31 to execute the color reverse conversion process. The color inverse conversion unit 31 converts the pixel signal S12 in the YUVK color space input from the post filter 32 into the pixel signal S13 in the RGrGbB color space based on the arithmetic expression shown in FIG. Further, the color reverse conversion unit 31 reproduces a RAW image having a honeycomb arrangement by applying an appropriate color to an appropriate position on the pixel plane based on the data D1 and D2. This makes it possible to appropriately reproduce the original image.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. It is a figure which shows the RAW image acquired by the image pick-up element of a honeycomb arrangement | sequence. It is a figure which shows the same RAW image as FIG. It is a figure which shows the 1st example of prescription | regulation of a pixel. It is a figure which shows the structure of a color conversion part roughly. It is a figure which shows the computing equation for a calculating part to perform a color conversion process by a calculation. It is a figure which shows the example of a parameter setting to a register. It is a figure which shows the conversion formula from RGrGbB color space to YUVK color space in HD Photo. It is a figure which shows the 2nd example of prescription | regulation of a pixel. It is a figure which shows the example of a parameter setting to a register. It is a block diagram which shows a part of structure of the image processing apparatus which concerns on the 2nd Embodiment of this invention. It is a figure which shows the 1st example of the register | resistor which a color conversion part has. It is a figure which shows the 2nd example of the register | resistor which a color conversion part has. It is a figure which shows the modification of 2nd Embodiment. It is a block diagram which shows a part of structure of the image processing apparatus which concerns on the 3rd Embodiment of this invention. It is a figure which shows roughly the 1st example of a structure of a conversion process part. It is a figure which shows roughly the 1st example of a structure of a conversion process part. It is a figure which shows schematically the 2nd example of a structure of a conversion process part. It is a figure which shows schematically the 2nd example of a structure of a conversion process part. It is a figure which shows the 1st example of prescription | regulation of a pixel. It is a block diagram which shows a part of structure of the image processing apparatus which concerns on the 4th Embodiment of this invention. It is a figure which shows the 2nd example of prescription | regulation of a pixel. It is a figure which shows the 3rd example of prescription | regulation of a pixel. It is a figure which shows the pixel signal output from a color conversion part. It is a figure which shows the computing equation for a color reverse conversion part to perform a color reverse conversion process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Encoder 3 Decoder 10 Image pick-up element 13,13A, 13B calculating part 15 Rearrangement process part 21 Color conversion part 31 Color reverse conversion part 40 Selection part 41A, 41B Conversion process part 50 Complementary process part

Claims (7)

A color conversion unit that converts a signal in the first color space acquired by the image pickup device having the honeycomb arrangement into a signal in the second color space based on a predetermined arithmetic expression;
The color conversion unit defines pixels including cells of a plurality of colors in the honeycomb arrangement, and performs a color conversion process in units of pixels.
The pixel that is the processing target of the color conversion unit includes a first pixel that is a first arrangement pattern and a second pixel that is a second arrangement pattern in the arrangement of cells of each color in the pixel. And
In the arithmetic expression, the signal value of each cell can be set as a parameter,
The color conversion unit sets the first parameter when processing the first pixel, and sets the signal value of the cell of each color in the arithmetic expression when processing the second pixel. An image processing apparatus, wherein a second parameter is set such that a position is equal to that of the first pixel. A color conversion unit that converts a signal in the first color space acquired by the image sensor with the honeycomb arrangement into a signal in the second color space;
A rearrangement processing unit that rearranges the signals of the first color space before the signals of the first color space are input to the color conversion unit;
The color conversion unit defines pixels including cells of a plurality of colors in the honeycomb arrangement, and performs a color conversion process in units of pixels.
The pixel that is the processing target of the color conversion unit includes a first pixel that is a first arrangement pattern and a second pixel that is a second arrangement pattern in the arrangement of cells of each color in the pixel. And
The rearrangement processing unit does not execute the rearrangement processing when the processing target of the color conversion unit is the first pixel, and the processing target of the color conversion unit is the second pixel. The image processing apparatus executes rearrangement processing so that the position of the signal value of the cell of each color in the pixel is equal to that of the first pixel. A color conversion unit that converts a signal in the first color space acquired by the image sensor with the honeycomb arrangement into a signal in the second color space;
A selection unit arranged in front of the color conversion unit,
The color conversion unit defines pixels including cells of a plurality of colors in the honeycomb arrangement, and performs a color conversion process in units of pixels.
The pixel that is the processing target of the color conversion unit includes a first pixel that is a first arrangement pattern and a second pixel that is a second arrangement pattern in the arrangement of cells of each color in the pixel. And
The color converter is
A first conversion processing unit that executes color conversion processing based on the first arithmetic expression;
And a second conversion processing unit that executes color conversion processing based on the second arithmetic expression such that the position of the signal value of each color cell is equivalently equal to that in the first arithmetic expression. And
The selection unit selects the first conversion processing unit when the processing target of the color conversion unit is the first pixel, and the processing target of the color conversion unit is the second pixel. In this case, the image processing apparatus selects the second conversion processing unit. A color conversion unit that converts a signal in the first color space acquired by the image sensor with the honeycomb arrangement into a signal in the second color space;
The color conversion unit defines pixels including cells of a plurality of colors in the honeycomb arrangement, and performs a color conversion process in units of pixels.
The color conversion unit is an image processing apparatus that defines pixels so that the arrangement of cells of each color in a pixel is a single arrangement pattern. For a pixel in which the number of cells in one pixel is insufficient at the periphery of the pixel plane, the image processing apparatus further includes a complement processing unit that complements a signal value related to the insufficient cell,
The image processing apparatus according to claim 4, wherein the complement processing unit complements the signal value related to the insufficient cell by a calculation using a signal value related to a cell located in the vicinity of the insufficient cell. For a pixel in which the number of cells in one pixel is insufficient at the periphery of the pixel plane, the image processing apparatus further includes a complement processing unit that complements a signal value related to the insufficient cell,
The image processing apparatus according to claim 4, wherein the complement processing unit supplements a signal value related to the insufficient cell using a predetermined value.   The color conversion unit outputs a signal of a second color space in which information on the definition of the pixel and information on the position of each color cell in the pixel are included in the header unit. The image processing apparatus according to any one of the above.

Images