JP2017200199A - Video compression device, video decoding device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device for suppressing a code amount during compression by color difference decimation.SOLUTION: In an image processing engine of an electronic camera, a compression processing unit 112 comprises a first decimation unit 13, a second decimation unit 14, a selection unit 12 and an encoding unit 16. The first decimation unit decimates color difference components from a color image of a first form in a first pattern, to generate a color image of a second form different from the first form. The second decimation unit decimates color difference components from a color image of the first form in a second pattern different from the first pattern, to generate a color image of the second form. The selection unit distributes a plurality of color images of the first form, inputted in time series, to the first decimation unit and the second decimation unit. The encoding unit encodes a plurality of color images of the second form, generated in the first decimation unit or the second decimation unit, along the time series.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a moving image compression apparatus, a moving image decoding apparatus, and a program.

  In moving image compression, for example, paying attention to the difference in visual sensitivity between luminance and color difference as in the YUV 4: 2: 0 format, the amount of information is reduced by thinning out the color difference component when compressing and encoding moving images. Efficient reduction has been conventionally performed.

  On the other hand, in YUV 4: 2: 0 format video, for example, when the video is enlarged by editing or the like, color blur tends to occur at the edge portion of the image. Therefore, a method for efficiently compressing and encoding a moving image without thinning out color difference components in the YUV 4: 4: 4 format has also been proposed (for example, Patent Document 1).

JP-A-6-276511

  However, when compressing and coding a moving image without thinning out the color difference component, color bleeding at the edge portion of the image can be suppressed at the time of decoding, but the amount of code at the time of compression increases compared to the case of thinning out the color difference component. End up.

  A moving image compression apparatus as an example of the present invention includes a first thinning unit, a second thinning unit, a selection unit, and an encoding unit. The first thinning unit thins out the color difference component from the color image of the first format with the first pattern, and generates a color image of the second format different from the first format. The second thinning unit thins out the color difference component from the first type color image with a second pattern different from the first pattern, and generates a second type color image. The selection unit distributes a plurality of first-format color images input in time series to the first thinning unit or the second thinning unit. The encoding unit encodes a plurality of second-format color images generated by the first thinning unit or the second thinning unit in time series.

  The moving image decoding apparatus includes an acquisition unit that acquires compressed moving image information, a decoding unit, and an interpolation unit. The compressed moving image information includes first encoded information obtained by encoding the first thinned image and second encoded information obtained by encoding the second thinned image. The first thinned image is an image converted from a color image of the first format into a color image of the second format different from the first format by thinning out the color difference component with the first pattern. The second thinned image is an image converted into a second format color image by thinning out the color difference component from the first format color image with a second pattern different from the first pattern. The decoding unit decodes the first thinned image from the first encoded information, and decodes the second thinned image from the second encoded information. The interpolation unit performs color difference interpolation processing on the first thinned image using the color difference component information of the second thinned image and outputs a color image of the first format. In addition, the interpolation unit performs color difference interpolation processing on the second thinned image using information on the color difference component of the first thinned image and outputs a color image of the first format.

  According to the example of the present invention, the code amount at the time of compression can be suppressed by the color difference thinning.

The figure which shows the structural example of the electronic camera of 1st Embodiment. The figure which shows the structural example of the compression process part of FIG. (A) The figure which shows the example of the sample point before color difference thinning out, (b) The figure which shows the example of the sample point in the 1st thinning image, (c) The figure which shows the example of the sample point in the 2nd thinning image, d) A diagram showing a state in which sample points of color differences of the first thinned image and the second thinned image are overlaid. The figure which shows the structural example of the encoding part of FIG. The figure which shows the structural example of the decoding process part of FIG. The figure which shows the structural example of the decoding part of FIG. The figure which shows the structural example of the interpolation part of FIG. Flow chart showing an example of moving image compression operation in the compression processing unit FIG. 8 is a flowchart showing an example of the encoding process at # 103 and # 107 in FIG. (A)-(c): The figure which shows the example of the encoding format of a moving image FIG. 9 is a flowchart showing an example of motion compensation prediction processing in # 205 of FIG. The figure which shows the example of correction | amendment of the motion vector of a color difference component Flow chart showing an example of a moving image decoding operation in the decoding processing unit FIG. 13 is a flowchart showing an example of the decoding process in # 401 of FIG. FIG. 14 is a flowchart showing an example of motion compensation prediction processing in # 507 of FIG. FIG. 13 is a flowchart showing an example of color difference interpolation processing in # 402 of FIG. The figure which shows the sample point of the color difference component interpolated by the 1st color difference interpolation part The figure which shows the example of the sample point after the color difference interpolation in a 1st color difference interpolation part The figure which shows the sample point of the color difference component interpolated by the 2nd color difference interpolation part The figure which shows the example of the sample point after the color difference interpolation in a 2nd color difference interpolation part The figure which shows the example of the local area | region when calculating | requiring the sample point C34 The figure which shows the example of the color difference channel in the output image of an addition part The figure which shows the apparatus structural example in 2nd Embodiment. The figure which shows another example of the encoding format of a moving image The figure which shows another example of the interpolation part

<Description of First Embodiment>
FIG. 1 is a diagram illustrating a configuration example of an electronic camera according to the first embodiment which is an example of an imaging device, a moving image compression device, and a moving image decoding device. Note that the electronic camera 100 of the first embodiment has a moving image shooting function.

  The electronic camera 100 includes an imaging optical system 101, an imaging element 102, a signal processing unit 103, an image processing engine 104, a first memory 105 and a second memory 106, a recording I / F 107, a monitor 108, and an operation. Part 109. Here, the signal processing unit 103, the first memory 105, the second memory 106, the recording I / F 107, the monitor 108, and the operation unit 109 are each connected to the image processing engine 104. The operation unit 109 is a switch that receives a user operation (for example, an instruction for moving image shooting, an instruction for mode switching, or the like).

  The imaging optical system 101 includes a plurality of lenses including, for example, a zoom lens and a focus lens. For simplicity, the imaging optical system 101 is illustrated as a single lens in FIG.

  The image sensor 102 is a device that images (captures) an image of a subject formed by a light beam that has passed through the imaging optical system 101. Note that in the moving image shooting mode, which is one of the shooting modes of the electronic camera 100, the image sensor 102 performs shooting of a moving image with recording in a nonvolatile storage medium (110). The output of the image sensor 102 is connected to the image processing engine 104. The above-described image sensor 102 may be a progressive scanning solid-state image sensor (for example, CCD) or an XY address type solid-state image sensor (for example, CMOS).

  Here, a plurality of light receiving elements (pixels) are arranged in a matrix on the light receiving surface of the image sensor 102. A plurality of types of color filters that transmit light of different color components are arranged in the pixels of the image sensor 102 according to a predetermined color arrangement. Therefore, each pixel of the image sensor 102 outputs an electrical signal corresponding to each color component by color separation in the color filter. For example, in the first embodiment, red (R), green (G), and blue (B) color filters are periodically arranged on the light receiving surface according to a known Bayer array. Thereby, the image sensor 102 can acquire a color image at the time of photographing.

  The signal processing unit 103 performs analog signal processing (correlated double sampling, black level correction, etc.), A / D conversion processing, and digital signal processing (defective pixel correction, etc.) on the image signal input from the image sensor 102. ) In order. Data of the image (RAW image) output from the signal processing unit 103 is input to the image processing engine 104.

  The image processing engine 104 is a processor that comprehensively controls the operation of the electronic camera 100. For example, the image processing engine 104 controls autofocus (AF) and automatic exposure (AE) using an image signal input from the image sensor 102.

  The image processing engine 104 includes an image processing unit 111, a compression processing unit 112, and a decoding processing unit 113.

  The image processing unit 111 performs, for example, color interpolation processing, gradation conversion processing, white balance adjustment processing, color conversion processing, and the like on the RAW image. By the color interpolation process described above, a RAW image in which each color is arranged in a mosaic pattern in a Bayer array is converted into an RGB image having RGB information at each pixel. In addition, the RGB image after color interpolation is converted into a color image (YUV image) in the YUV color space by the color conversion process described above. The YUV image output from the image processing unit 111 is an image in the YUV 4: 4: 4 format, and each pixel has information on a luminance component (Y) and two color difference components (U, V). Yes.

  The compression processing unit 112 is an example of a moving image compression device, and compresses and encodes the YUV image output from the image processing unit 111. A configuration example of the compression processing unit 112 will be described later.

  The decoding processing unit 113 is an example of a moving image decoding device, and decodes moving image data compression-encoded by the compression processing unit 112. A configuration example of the decoding processing unit 113 will be described later.

  The functional blocks of the image processing unit 111, the compression processing unit 112, and the decoding processing unit 113 included in the image processing engine 104 can be realized by an arbitrary processor, memory, or other LSI in terms of hardware, and in terms of software This is realized by a program loaded in the memory.

  The first memory 105 temporarily stores image data in the pre-process and post-process of image processing. For example, the first memory 105 is an SDRAM that is a volatile storage medium. The second memory 106 stores a program executed by the image processing engine 104 and various data used by this program. For example, the second memory 106 is a nonvolatile memory such as a flash memory.

  The recording I / F 107 has a connector for connecting the nonvolatile storage medium 110. The recording I / F 107 writes / reads moving image data to / from the storage medium 110 connected to the connector. The storage medium 110 is a hard disk, a memory card incorporating a semiconductor memory, or the like. In FIG. 1, a memory card is illustrated as an example of the storage medium 110.

  The monitor 108 is a display device that displays various images. For example, the monitor 108 controls the image processing engine 104 to display a viewfinder at the time of moving image shooting and to reproduce and display a captured moving image.

  Next, a configuration example of the compression processing unit 112 in FIG. 1 will be described with reference to FIG. In FIG. 2, each unit of the compression processing unit 112 is shown as a functional block realized by cooperation of hardware and software.

  The compression processing unit 112 includes a filter processing unit 11, a first switch 12, a first thinning unit 13, a second thinning unit 14, a second switch 15, an encoding unit 16, and a compression control unit 17.

  The filter processing unit 11 performs a two-dimensional low-pass filter process on the color difference component of the input YUV image. The output of the filter processing unit 11 is connected to the first switch 12.

  For example, the filter processing unit 11 performs through output without performing processing on the Y channel (Y plane) in the YUV image. On the other hand, the filter processing unit 11 applies a two-dimensional low-pass filter process to the U channel (U plane) and V channel (V plane) of the YUV image and outputs the result. As a result, for example, occurrence of aliasing due to color difference thinning described later is suppressed. The above-described low-pass filter is selected in accordance with the color difference thinning characteristics described later, and a diamond-type low-pass filter is applied as an example.

  The first switch 12 is an example of a selection unit, and is a 1-input 2-output switch. One output of the first switch 12 is connected to the first decimation unit 13, and the other output of the first switch is connected to the second decimation unit 14. The first switch 12 distributes the plurality of YUV images input in time series from the filter processing unit 11 to the first thinning unit 13 or the second thinning unit 14. In the first embodiment, the first switch 12 alternately outputs each frame of the moving image of the YUV image to the first thinning unit 13 and the second thinning unit 14.

  The first thinning unit 13 performs a color difference thinning process for thinning the color difference components from the YUV image using the first pattern. In the following description, an image that has been subjected to color difference thinning by the first thinning unit 13 may be referred to as a first thinned image.

  For example, the first thinning unit 13 discretely extracts odd-numbered sample points every other pixel along the row direction in the U channel and V channel of the YUV image. Then, the first decimation unit 13 outputs only the extracted sample point information to the subsequent stage for the U channel and the V channel. Therefore, in the color difference thinning of the first thinning unit 13, the U channel and V channel sample points are thinned to ¼. On the other hand, the first thinning-out unit 13 outputs the Y channel information of the YUV image through without performing thinning. As a result, the first thinned image is converted into the YUV 4: 2: 0 format.

  Here, FIG. 3A is a diagram illustrating an example of sample points before color difference thinning. FIG. 3B is a diagram illustrating an example of sample points in the first thinned image. In the example of FIG. 3, the sample point of luminance (Y) and the sample point of color difference (U, V) are shown superimposed.

  The second thinning unit 14 performs a color difference thinning process for thinning out the color difference component from the YUV image with a second pattern different from the first pattern. In the following description, an image that has been subjected to color difference thinning by the second thinning unit 14 may be referred to as a second thinned image.

  For example, the second thinning unit 14 discretely extracts even-row sample points every other pixel along the row direction in the U channel and V channel of the YUV image. Here, the position of the color difference sample point extracted by the second thinning unit 14 is shifted by one pixel in the row direction (X direction) with respect to the position of the color difference sample point extracted by the first thinning unit 13. Yes. Then, for the U channel and V channel, the second thinning unit 14 outputs only the information of the extracted sample points to the subsequent stage. Therefore, in the color difference thinning of the second thinning unit 14, the U channel and V channel sample points are thinned to ¼. On the other hand, the second thinning unit 14 outputs the Y channel information of the YUV image through without performing thinning. As a result, as in the case of the first thinned image, the second thinned image is converted into the YUV 4: 2: 0 format.

  Here, FIG. 3C is a diagram illustrating an example of sample points in the second thinned image. FIG. 3D is a diagram showing a state in which sample points of color differences of the first thinned image and the second thinned image are overlapped.

  As described above, in the color difference thinning of the first thinning-out unit 13, the odd-numbered color difference sampling points are extracted every other pixel. Extracted. The position of the color difference sample point extracted by the second thinning unit 14 is shifted by one pixel in the row direction (X direction) with respect to the position of the color difference sample point extracted by the first thinning unit 13. . Therefore, the color difference sample points extracted by the first decimation unit 13 and the color difference sample points extracted by the second decimation unit 14 are in a positional relationship that does not overlap each other in the spatial direction. When the positions of these sample points are overlapped, the color difference sample points are arranged in a checkered pattern on the image (see FIG. 3D).

  The second switch 15 is a two-input one-output switch corresponding to the first switch 12. One input of the second switch 15 is connected to the first decimation unit 13, and the other output of the second switch 15 is connected to the second decimation unit 14. The output of the second switch 15 is connected to the encoding unit 16. The second switch 15 operates in conjunction with the first switch 12. That is, the second switch 15 is connected to the first thinning unit 13 when the first switch 12 is connected to the first thinning unit 13, and the second switch 15 is connected to the second thinning unit 14. The switch 15 is connected to the second thinning unit 14. Accordingly, the first thinned image output from the first thinning unit 13 and the second thinned image output from the second thinning unit 14 are alternately input to the encoding unit 16 along the time series of the moving images. Is done.

  The encoding unit 16 compresses and encodes a YUV 4: 2: 0 format moving image input in time series. For example, the encoding unit 16 divides each frame of the moving image into blocks of a predetermined size, and compresses and encodes the moving image data using motion compensation prediction. A configuration example of the encoding unit 16 will be described later.

  In the following description, information obtained by encoding the first thinned image is also referred to as first encoded information, and information obtained by encoding the second thinned image is also referred to as second encoded information. The moving image data compressed by the compression processing unit 112 may be referred to as compressed moving image information.

  The compression control unit 17 unifies the operations of the filter processing unit 11, the first switch 12, the first thinning unit 13, the second thinning unit 14, the second switch 15, and the encoding unit 16 when compressing a moving image. Control. Note that, under the control of the compression control unit 17, the encoding unit 16 adds incidental information indicating a color difference thinning pattern to, for example, a header region of an encoded image.

  Next, a configuration example of the encoding unit 16 in FIG. 2 will be described with reference to FIG. In FIG. 4, each part of the encoding part 16 is shown by the functional block implement | achieved by cooperation of hardware or software.

  Here, as an image compression encoding method for performing motion compensation prediction, for example, MPEG-2 video defined in ISO / IEC13818-2 is known. In the above-described image compression coding system, compression coding is performed within a frame or between frames using I pictures, P pictures, and B pictures. An I picture is an intra-picture encoded image obtained by encoding that is completed only within the picture. A P picture is an inter-screen predictive encoded image obtained by predictive encoding using a maximum of one reference image. A B picture is an inter-screen bi-predictive encoded image obtained by predictive encoding using a maximum of two reference images.

  The encoding unit 16 includes a first accumulation unit 21, a subtraction unit 22, an orthogonal transformation unit 23, a quantization unit 24, a variable length coding unit 25, an inverse quantization unit 26, an inverse orthogonal transformation unit 27, an addition unit 28, 2 It has the storage part 29, the prediction information generation part 30, and the prediction part 31.

  The first accumulation unit 21 accumulates the image after color difference thinning out output from the second switch 15. The image stored in the first storage unit 21 is output to the subtraction unit 22 in the order of input as an image to be encoded. Note that the encoded image is sequentially deleted from the first storage unit 21.

  When generating a P picture or a B picture, the subtracting unit 22 outputs a difference signal (prediction error value) between the input original image and a prediction value described later generated by the prediction unit 31. Further, when generating the I picture, the subtracting unit 22 outputs the input original image signal as it is.

  When generating an I picture, the orthogonal transform unit 23 performs orthogonal transform on the signal of the original image input through the subtraction unit 22. Further, when generating the P picture or the B picture, the orthogonal transform unit 23 performs orthogonal transform on the above-described difference signal.

  The quantization unit 24 converts the frequency coefficient (orthogonal transform coefficient) of the block unit input from the orthogonal transform unit 23 into a quantized coefficient. The output of the quantization unit 24 is input to the variable length encoding unit 25 and the inverse quantization unit 26, respectively.

  The variable length coding unit 25 performs variable length coding on prediction information such as quantization coefficients and motion vectors, and outputs a coded bit stream of a moving image to the outside. Note that the variable-length encoding unit 25 indicates whether or not it is necessary to correct the motion vector with the color difference component when adding the above-described supplementary information or performing motion prediction between the first thinned image and the second thinned image. Correction information is also given.

  For example, when compression encoding is performed in accordance with the main profile of MPEG-2 video, the variable length encoding unit 25 may include the above-described supplementary information and correction information in the user data area following the picture coding extension.

  The inverse quantization unit 26 inversely quantizes the quantization coefficient in units of blocks and decodes the frequency coefficient. The inverse orthogonal transform unit 27 performs inverse orthogonal transform on the frequency coefficient decoded by the inverse quantization unit 26 to decode a prediction error value (or an original image signal).

  The adding unit 28 adds the decoded prediction error value and a prediction value described later generated by the prediction unit 31. The decoded value (reference image) of the picture output from the adding unit 28 is stored in the second storage unit 29. Note that images that are not referred to in subsequent motion compensation prediction are sequentially deleted from the second storage unit 29.

  The prediction information generation unit 30 uses the reference image of the second storage unit 29 to generate prediction information (such as a motion vector) for predicting the encoding target image. The prediction information is output to the prediction unit 31 and the variable length coding unit 25.

  The prediction unit 31 outputs a prediction value obtained by predicting the encoding target image in units of blocks based on the prediction information and the reference image. This predicted value is output to the subtractor 22 and the adder 28.

  Note that when motion compensation prediction is performed for a certain block, only prediction information is converted into data when the encoding target image completely matches the prediction value. Further, when the image to be encoded partially matches the predicted value, the prediction information and the difference image are converted into data. When all the encoding target images are out of the predicted values, all the images for the entire block are converted into data.

  Next, a configuration example of the decoding processing unit 113 in FIG. 1 will be described with reference to FIG. In FIG. 5, each unit of the decoding processing unit 113 is shown as a functional block realized by cooperation of hardware and software.

  The decoding processing unit 113 includes a decoding unit 41, an interpolation unit 42, and a decoding control unit 43. Note that the compressed moving image information is input to the decoding processing unit 113.

  The decoding unit 41 decodes the first thinned image from the first encoded information included in the compressed moving image information, and decodes the second thinned image from the second encoded information included in the compressed moving image information. The output of the decoding unit 41 is connected to the interpolation unit 42. A configuration example of the decoding unit 41 will be described later.

  The interpolation unit 42 performs color difference interpolation processing on the U channel and the V channel of the first thinned image or the second thinned image in the YUV 4: 2: 0 format. Then, the interpolating unit 42 outputs through without performing processing on the Y channel (Y plane) of the first thinned image or the second thinned image. As a result, the interpolation unit 42 outputs a color image in the YUV 4: 4: 4 format.

  The decoding control unit 43 comprehensively controls each operation of the decoding unit 41 and the interpolation unit 42 when decoding a moving image. Note that, under the control of the decoding control unit 43, the interpolation unit 42 refers to the header area of the image and acquires incidental information indicating a color difference thinning pattern. Then, the interpolation unit 42 determines a combination of images to be used for the color difference interpolation process based on the accompanying information.

  Next, a configuration example of the decoding unit 41 in FIG. 5 will be described with reference to FIG. In FIG. 6, each part of the decoding part 41 is shown by the functional block implement | achieved by cooperation of hardware or software. Note that the functional block of the decoding unit 41 shown in FIG. 6 may be shared with the functional block of the same name of the encoding unit 16 shown in FIG.

  The decoding unit 41 includes a code decoding unit 51, an inverse quantization unit 52, an inverse orthogonal transform unit 53, an addition unit 54, a first accumulation unit 55, a second accumulation unit 56, and a prediction unit 57.

  The code decoding unit 51 decodes an encoded bit stream of an input moving image and outputs prediction information such as a quantization coefficient and a motion vector. The decoded quantization coefficient is input to the inverse quantization unit 52, and the decoded prediction information is input to the prediction unit 57. Note that when the compressed moving image information includes correction information indicating whether or not the motion vector needs to be corrected with the color difference component, the code decoding unit 51 inputs the decoded correction information to the prediction unit 57.

  The inverse quantization unit 52 performs inverse quantization on the quantization coefficient in units of blocks and decodes the frequency coefficient. The inverse orthogonal transform unit 53 performs inverse orthogonal transform on the frequency coefficient decoded by the inverse quantization unit 52 to decode a prediction error value (or an original image signal).

  The adding unit 54 adds the decoded prediction error value and the prediction value generated by the prediction unit 57 to output a decoded image in units of blocks. The decoded value of the image output from the adding unit 54 is input to the first storage unit 55 and the second storage unit 56, respectively.

  The first accumulation unit 55 accumulates the decoded first thinned image or second thinned image until it is output to the interpolation unit 42. Note that images that have been output to the interpolation unit 42 are sequentially deleted from the first storage unit 55. The second storage unit 56 stores the decoded image value as a reference image. Note that images that are not referred to in subsequent motion compensation prediction are sequentially deleted from the second storage unit 56.

  The prediction unit 57 outputs a prediction value obtained by predicting the decoding target image in units of blocks based on the prediction information and the reference image to the addition unit 54. The prediction unit 57 corrects the motion vector of the color difference component according to the difference in the thinning pattern when the color difference component thinning pattern differs between the decoding target image and the prediction source image.

  Next, a configuration example of the interpolation unit 42 in FIG. 5 will be described with reference to FIG. In FIG. 7, each part of the interpolation part 42 is shown by the functional block implement | achieved by cooperation of hardware or software.

  The interpolation unit 42 includes a storage unit 61, a first color difference interpolation unit 62, a color difference synthesis unit 63, a second color difference interpolation unit 64, a weight adjustment unit 65, a first multiplication unit 66, a second multiplication unit 67, and an addition unit 68. doing.

  The accumulation unit 61 temporarily accumulates the first and second thinned images used in the color difference interpolation process. For example, in the color difference interpolation process, the first thinned image and the second thinned image that are adjacent in the time axis direction are used, but images that are no longer used in the color difference interpolation process are sequentially deleted from the storage unit 61.

  The first chrominance interpolation unit 62 interpolates the chrominance components thinned out in the image serving as a reference for the chrominance interpolation within the frame without using the chrominance component information of the different frames, and the image in the YUV 4: 4: 4 format. Is output. The output of the first color difference interpolation unit 62 is connected to the first multiplication unit 66.

  The color difference synthesizing unit 63 synthesizes the color difference components of two images adjacent to each other in the time axis direction with different color difference thinning patterns. The color difference synthesis unit 63 replenishes the color difference components in a row in which the color difference components are completely thinned out from an image serving as a reference for color difference interpolation from adjacent frames in the time axis direction. Note that the color difference synthesis unit 63 supplements the color difference component from an image one frame past, for example, with respect to an image serving as a reference for color difference interpolation.

  For example, when the reference for color difference interpolation is the first thinned image, the sample points of the color difference components in even rows are supplemented from the second thinned image adjacent in the time axis direction. When the reference for color difference interpolation is the second thinned image, the sample points of the odd-numbered color difference components are supplemented from the first thinned image adjacent in the time axis direction. Thereby, in the image output from the color difference synthesis unit 63, the sample points of the color difference components are arranged in a checkered pattern (see FIG. 3D).

  The second color difference interpolation unit 64 interpolates the color difference components of the image output from the color difference synthesis unit 63, and outputs a YUV 4: 4: 4 format image. Note that the output of the second color difference interpolation unit 64 is connected to the second multiplication unit 67.

  The weight adjustment unit 65 obtains weighting coefficients α and 1-α when adding the color difference component of the first color difference interpolation unit 62 and the color difference component of the second color difference interpolation unit 64 (where 0 ≦ α ≦ 1). Here, the weight adjustment unit 65 changes the above-described coefficients α and 1-α for each sample point of the image. For example, the weight adjustment unit 65 obtains the above-described coefficients α and 1-α for each sample point according to the luminance component of the local region corresponding to the sample point.

  The first multiplication unit 66 multiplies the color difference component of the first color difference interpolation unit 62 by a coefficient α. The second multiplier 67 multiplies the color difference component of the second color difference interpolation unit 64 by a coefficient 1-α. Further, the adding unit 68 adds the color difference component of the first multiplication unit 66 weighted with the coefficient α and the color difference component of the second multiplication unit 67 weighted with the coefficient 1−α for each sample point. Then, the adding unit 68 outputs an image in YUV 4: 4: 4 format subjected to color difference interpolation to the outside.

(Example of video compression)
Hereinafter, an example of a moving image compression operation in the first embodiment will be described.

  When the electronic camera 100 according to the first embodiment receives a shooting instruction in the moving image shooting mode, the image processing engine 104 drives the image sensor 102 to start moving image shooting. A frame of a moving image output from the image sensor 102 is subjected to a series of signal processing and image processing by the signal processing unit 103 and the image processing unit 111, and then compression encoded by the compression processing unit 112. The compressed moving image information compressed by the compression processing unit 112 is recorded on the storage medium 110 connected to the recording I / F 107 under the control of the image processing engine 104.

  FIG. 8 is a flowchart showing an example of a moving image compression operation in the compression processing unit 112.

  Step # 101: The filter processing unit 11 of the compression processing unit 112 performs a two-dimensional low-pass filter process on the color difference component of the input YUV image.

  Step # 102: The compression control unit 17 switches the first switch 12 and the second switch 15 to the first decimation unit 13 side. Then, the first thinning unit 13 performs a color difference thinning process for thinning the color difference components from the YUV image with the first pattern. As a result, a first thinned image in the YUV 4: 2: 0 format is generated (see FIG. 3B).

  Step # 103: The encoding unit 16 compresses and encodes the input first thinned image in accordance with, for example, the main profile of MPEG-2 video. The first encoded information after compression is recorded in the storage medium 110.

  Step # 104: The compression control unit 17 determines whether or not the compression of the entire input moving image has been completed. If the above requirement is satisfied (YES side), the compression control unit 17 ends the compression operation. On the other hand, if the above requirement is not satisfied (NO side), the process proceeds to # 105 and the next frame is compressed.

  Step # 105: The filter processing unit 11 performs a two-dimensional low-pass filter process on the color difference component of the input YUV image.

  Step # 106: The compression control unit 17 switches the first switch 12 and the second switch 15 to the second decimation unit 14 side. Then, the second thinning unit 14 performs a color difference thinning process for thinning the color difference components from the YUV image with the second pattern. As a result, a second thinned image in YUV 4: 2: 0 format is generated (see FIG. 3C).

  Step # 107: The encoding unit 16 compresses and encodes the input second thinned image in accordance with, for example, the main profile of MPEG-2 video. The compressed second encoded information is recorded in the storage medium 110.

  Step # 108: The compression control unit 17 determines whether or not the compression of the entire input moving image has been completed. If the above requirement is satisfied (YES side), the compression control unit 17 ends the compression operation. On the other hand, when the above requirement is not satisfied (NO side), the process returns to # 101 to compress the next frame. Above, description of FIG. 8 is complete | finished.

  FIG. 9 is a flowchart showing an example of the encoding process at # 103 and # 107 in FIG.

  Step # 201: The compression control unit 17 determines an encoding format (I picture, P picture, B picture) of an image to be encoded.

  Here, FIGS. 10A to 10C are diagrams illustrating examples of the encoding format of the moving image. In FIG. 10, “In” indicates the nth I picture, “Pn” indicates the nth P picture, and “Bn” indicates the nth B picture. In the example of FIG. 10, an odd-numbered frame is a first thinned image, and an even-numbered frame is a second thinned image.

  FIG. 10A shows an example in which each frame of a moving image is encoded as an I picture. In the case of FIG. 10A, only intra prediction encoding is performed, and inter prediction encoding is not performed between the first thinned image and the second thinned image.

  FIG. 10B shows an example in which a moving image is encoded with a combination of an I picture and a P picture. In the case of FIG. 10B, inter-screen predictive coding is performed between the first thinned image and the second thinned image.

  FIG. 10C illustrates an example in which a moving image is encoded with a combination of an I picture, a P picture, and a B picture. In the case of FIG. 10C, the I picture and P picture are odd-numbered frames (first thinned images), and the B picture is an even-numbered frame (second thinned images). In the coding of the P picture in FIG. 10C, the inter-picture prediction coding is performed with reference to only images having the same color difference thinning pattern. On the other hand, in the coding of the B picture in FIG. 10C, the inter-screen bi-predictive coding is performed on the second thinned image with reference to the first thinned images adjacent to each other in the time axis direction.

  Step # 202: The compression controller 17 identifies the color difference thinning pattern (first thinned image, second thinned image) of the image to be encoded.

  Step # 203: The compression control unit 17 determines whether or not it is necessary to correct the motion vector of the color difference component in the motion compensation prediction at the time of compression according to the color difference thinning pattern between the image to be encoded and the reference image. To do.

  For example, when inter-picture predictive encoding is performed using an image with the same color difference thinning pattern as a reference image, the positions of the sample points match between the image to be encoded and the reference image. However, when inter-picture predictive encoding is performed using an image with a different color difference thinning pattern as a reference image, the sample point is displaced between the image to be encoded and the reference image. Therefore, in the latter case (for example, in the case of the P picture in FIG. 10B and the B picture in FIG. 10C), the compression control unit 17 performs the motion vector of the color difference component in the motion compensation prediction. Set to correct.

  Step # 204: Under the control of the compression control unit 17, the variable length coding unit 25 of the coding unit 16 adds the additional information and the correction information to the header area of the image to be coded, and performs coding. The incidental information is, for example, an identifier indicating the first thinned image and the second thinned image. The correction information is, for example, an identifier indicating whether correction is necessary for the motion vector of the color difference component described above.

  For example, when compression encoding is performed in accordance with the main profile of MPEG-2 video, the variable length encoding unit 25 includes the above-described supplementary information and correction information in the user data area following the picture coding extension.

  Step # 205: When generating the P picture or the B picture, the prediction information generation unit 30 performs motion compensation prediction on the macroblock using the reference image. Further, when generating an I picture, the prediction information generation unit 30 outputs the macroblock information as a through output.

  Step # 206: When generating the P picture or the B picture, the prediction unit 31 outputs the prediction value of the macroblock obtained by the motion compensation prediction (# 205). Further, when generating the I picture, the prediction unit 31 outputs a zero value as a prediction value. Then, the subtraction unit 22 subtracts the prediction value from the original signal of the image and outputs a prediction error.

  Step # 207: The orthogonal transform unit 23 performs orthogonal transform on the prediction error (# 206).

  Step # 208: The quantization unit 24 quantizes the orthogonal transform coefficient (# 207) of the prediction error.

  Step # 209: The variable length coding unit 25 performs variable length coding on prediction information indicating a motion vector, a reference image, and the like, and a quantization coefficient. In step # 209, the encoding information for one block is output from the encoding unit 16.

  Step # 210: The inverse quantization unit 26 inversely quantizes the quantization coefficient.

  Step # 211: The inverse orthogonal transform unit 27 performs inverse orthogonal transform on the orthogonal transform coefficient obtained by inverse quantization and decodes the prediction error.

  Step # 212: The adding unit 28 adds the predicted value of the macroblock to the decoded prediction error to generate a local decoded value of the picture. Note that the above-described local decoded value is stored in the second storage unit 29.

  Step # 213: The compression control unit 17 determines whether or not one frame has been encoded. If the above requirement is satisfied (YES side), the process returns to # 104 or # 108. On the other hand, when the above requirement is not satisfied (NO side), the processing returns to # 205 and the encoding unit 16 repeats the above operation. By the loop described above, the image to be encoded is encoded in units of blocks. Above, description of FIG. 9 is complete | finished.

  FIG. 11 is a flowchart showing an example of the motion compensation prediction process in # 205 of FIG.

  Step # 301: The prediction information generation unit 30 generates prediction information such as a macroblock motion vector, a prediction mode, and a reference image.

  Step # 302: The prediction information generation unit 30 determines whether or not the encoding format is an I picture. If the above requirement is satisfied (YES side), the process returns to # 206 in FIG. In this case, the prediction information generation unit 30 outputs the macroblock information as a through output. On the other hand, if the above requirement is not satisfied (NO side), the process proceeds to # 303.

  Step # 303: The prediction information generation unit 30 performs motion compensation prediction on the luminance component of the macroblock based on the above-described prediction information.

  Step # 304: The prediction information generation unit 30 determines whether or not the setting is to correct the motion vector of the color difference component based on the process of # 203 in FIG. If the above requirement is satisfied (YES side), the process proceeds to # 305. On the other hand, if the above requirement is not satisfied (NO side), the process proceeds to # 307. In the case of # 304 on the NO side, motion compensation prediction is performed for the color difference component without correcting the motion vector of the color difference component.

  Step # 305: The prediction information generating unit 30 determines whether or not the color difference thinning pattern is different between the encoding target image and the reference image. If the above requirement is satisfied (YES side), the process proceeds to # 306. On the other hand, if the above requirement is not satisfied (NO side), the process proceeds to # 307. Note that in the case of # 305 on the NO side, motion compensation prediction is performed for the color difference component without correcting the motion vector of the color difference component.

  Step # 306: The prediction information generation unit 30 corrects the motion vector of the color difference component according to the difference in the color difference thinning pattern from the reference image.

  FIG. 12 is a diagram illustrating a correction example of the motion vector of the color difference component. FIG. 12 shows the value of the color difference component at the target pixel P2x of the prediction destination image (second thinned image) and the average value of the four pixels (P1a to P1d) of the prediction source image (first thinned image) adjacent to the target pixel. Shows an example to be obtained. In the case of FIG. 12, since the pixel value of the target pixel is predicted by the average of four adjacent pixels, the encoding distortion included in the decoded value of the prediction source image can be suppressed.

  Regarding the following description, in FIG. 12, the arrangement of the color difference component sample points (pixels P) is indicated by circles on the image. Further, the positional deviation of the reference pixel of the prediction source image with respect to the position of the target pixel of the prediction destination image is referred to as a motion vector MV [x, y]. In the motion vector MV, x increases due to a right shift, and x decreases due to a left shift. Similarly, in the motion vector MV, y increases due to a downward shift, and y decreases due to an upward shift.

  In the example of FIG. 12, P1 pixels are arranged at the upper left of the color array, and P2 pixels are arranged at the lower right of the color array. In the case of FIG. 12, the pixel P1a and the pixel P2x belong to the same sample point, but the target pixel (P2x) is affected by the pixels P1b to P1d by interpolation. Therefore, the range of the reference pixel in the prediction source image is shifted by 0.5 pixel on the right and 0.5 pixel on the lower side with respect to the position of the target pixel (P2x). Therefore, in the example of FIG. 12, the motion vector MV of the color difference component is [0.5, 0.5].

  The prediction information generation unit 30 in # 306 may correct the motion vector of the color difference component so as to cancel the motion vector MV.

  Step # 307: The prediction information generation unit 30 performs motion compensation prediction for the color difference component of the macroblock based on the above-described prediction information. Thereafter, the process returns to # 206 of FIG. This is the end of the description of FIG.

(Example of video decoding operation)
Hereinafter, an example of a moving image decoding operation according to the first embodiment will be described.

  When the electronic camera 100 according to the first embodiment receives an instruction to reproduce the compressed moving image information compressed by the compression processing unit 112, the image processing engine 104 reads the compressed moving image information from the storage medium 110 connected to the recording I / F 107. To get. Then, the decoding processing unit 113 decodes the compressed moving image information. The decoded moving image is reproduced and displayed, for example, on the monitor 108 under the control of the image processing engine 104.

  FIG. 13 is a flowchart illustrating an example of a moving image decoding operation in the decoding processing unit 113.

  Step # 401: The decoding processing unit 113 acquires compressed video information from the storage medium 110. Then, the decoding unit 41 decodes the thinned image for one frame from the encoded information.

  Step # 402: The interpolation unit 42 performs color difference interpolation processing on the decoded YUV 4: 2: 0 format thinned image, and outputs a YUV 4: 4: 4 format color image.

  Step # 403: The decoding control unit 43 determines whether or not decoding of the acquired compressed video information has been completed. When the above requirements are satisfied (YES side), the decoding control unit 43 ends the decoding operation. On the other hand, when the above requirement is not satisfied (NO side), the process returns to # 401 to decode the next frame. Above, description of FIG. 13 is complete | finished.

  FIG. 14 is a flowchart showing an example of the decoding process in # 401 of FIG.

  Step # 501: The encoding / decoding unit 51 decodes the information of the header area of the encoded image. As a result, the decoding control unit 43 acquires the above-described incidental information and correction information from the user data area following the picture coding extension.

  Step # 502: The decoding control unit 43 determines whether or not correction is required for the motion vector of the chrominance component in the motion compensation prediction at the time of decoding based on the above correction information. For example, when the motion vector of the color difference component is corrected at the time of compression, the decoding control unit 43 corrects the motion vector of the color difference component even in motion compensation prediction at the time of decoding.

  Step # 503: The code decoding unit 51 decodes a variable length code of a certain macroblock, and acquires prediction information indicating a motion vector, a reference image, and the like, and a quantization coefficient.

  Step # 504: The inverse quantization unit 52 inversely quantizes the quantization coefficient (# 503) of the macroblock.

  Step # 505: The inverse orthogonal transform unit 53 performs inverse orthogonal transform on the orthogonal transform coefficient obtained by inverse quantization (# 504) to decode a prediction error.

  Step # 506: When decoding a P picture or a B picture, the prediction unit 57 performs motion compensation prediction on the macroblock using the prediction information, and outputs a prediction value of the macroblock. Further, the prediction unit 57 outputs a zero value as a prediction value when decoding an I picture.

  Step # 507: The adding unit 54 adds the predicted value of the macroblock to the decoded prediction error to generate a local decoded value of the picture. As a result, one macroblock image is decoded. The above-described local decoded value is stored in the second storage unit 56.

  Step # 508: The decoding control unit 43 determines whether or not decoding of one frame has been completed. If the above requirement is satisfied (YES side), the process returns to # 402. On the other hand, when the above requirement is not satisfied (NO side), the process returns to # 503 and the decoding unit 41 repeats the above operation. According to the above loop, the encoded image is decoded in units of blocks. Above, description of FIG. 14 is complete | finished.

  FIG. 15 is a flowchart showing an example of motion compensation prediction processing in # 507 of FIG.

  Step # 601: The prediction unit 57 sets a macroblock motion vector, a prediction mode, a reference image, and the like based on the decoded prediction information (# 503).

  Step # 602: Based on the header area information (# 501) and the prediction information (# 503), the prediction unit 57 performs the color difference thinning pattern (first thinned image, second thinned image) of the image to be decoded and the reference image. ) To identify each.

  Step # 603: The prediction unit 57 performs motion compensation prediction on the luminance component of the macroblock based on the above prediction information.

  Step # 604: The prediction unit 57 determines whether or not the setting is to correct the motion vector of the color difference component based on the process of # 502 in FIG. If the above requirement is satisfied (YES side), the process proceeds to # 605. On the other hand, if the above requirement is not satisfied (NO side), the process proceeds to # 607. In the case of # 604 on the NO side, motion compensation prediction is performed for the color difference component without correcting the motion vector of the color difference component.

  Step # 605: The prediction unit 57 determines whether or not the color difference thinning pattern is different between the decoding target image and the reference image. If the above requirement is satisfied (YES side), the process proceeds to # 606. On the other hand, if the above requirement is not satisfied (NO side), the process proceeds to # 607. In the case of # 605 on the NO side, motion compensation prediction is performed for the color difference component without correcting the motion vector of the color difference component.

  Step # 606: The prediction unit 57 corrects the motion vector of the color difference component according to the difference in the color difference thinning pattern from the reference image. The correction of the motion vector of the color difference component in # 606 is the reverse process of # 306 in FIG.

  Step # 607: The prediction unit 57 performs motion compensation prediction on the color difference component of the macroblock based on the above-described prediction information. Thereafter, the processing returns to # 506 in FIG. This is the end of the description of FIG.

  FIG. 16 is a flowchart showing an example of the color difference interpolation process in # 402 of FIG.

  Step # 701: The first chrominance interpolation unit 62 interpolates the chrominance components corresponding to the thinned out images in the image serving as the chrominance interpolation reference within the frame without using the chrominance component information of different frames.

  FIG. 17 is a diagram illustrating sample points of color difference components that are interpolated by the first color difference interpolation unit 62. As an example, consider a case where the color difference components of the sample points C34, C43, and C44 are interpolated in the thinned image shown in FIG. The sample point C33 is a color difference sample point included in the nth frame thinned image. At this time, the first color difference interpolation unit 62 may obtain the color difference component of the sample point to be interpolated by the calculation of the equations (1) to (4).

  Here, in the description of this specification, “p” indicates a color difference value obtained by intra-frame interpolation. “Cγ [n]” indicates the value of the color difference at the sample point γ of the nth frame.

  For example, when the first thinned image shown in FIG. 3B is subjected to color difference interpolation by the first color difference interpolation unit 62, the image after color difference interpolation is in the YUV 4: 4: 4 format as shown in FIG.

  Step # 702: The color difference synthesizing unit 63 synthesizes the color difference components of two images that have different color difference thinning patterns and are adjacent in the time axis direction. In the image output from the color difference synthesis unit 63, the sample points of the color difference components are arranged in a checkered pattern (see FIG. 3D).

  Step # 703: The second color difference interpolation unit 64 interpolates the color difference components of the image (FIG. 3D) output from the color difference synthesis unit 63.

  FIG. 19 is a diagram illustrating sample points of color difference components that are interpolated by the second color difference interpolation unit 64. As an example, consider the case of interpolating the color difference components of sample points C34 and C43 shown in FIG. The sample point C33 is a color difference sample point included in the nth frame thinned image, and the sample point C44 is a color difference sample point supplemented from the n−1th frame thinned image. At this time, the second color difference interpolation unit 64 may obtain the color difference component of the sample point to be interpolated by the calculation of Expressions (5) to (8).

  Here, in the description of the present specification, “q” indicates a color difference value obtained by inter-frame interpolation. “Sγ [n] h” indicates the horizontal correlation at the sampling point γ of the nth frame, and “Sγ [n] v” indicates the correlation in the vertical direction at the sampling point γ of the nth frame. Show. “Kγ [n] h” represents a horizontal weighting value at the sampling point γ in the nth frame, and “Kγ [n] v” represents a weighting in the vertical direction at the sampling point γ in the nth frame. Indicates the value.

  For example, when an image obtained by synthesizing the color difference sample points shown in FIG. 3D is interpolated by the second color difference interpolation unit 64, the image after the color difference interpolation is in the YUV 4: 4: 4 format as shown in FIG. Become.

  Step # 704: The weighting adjustment unit 65 obtains the above-described weighting coefficients α, 1-α for each sample point according to the luminance component of the local region corresponding to the sample point.

  For example, the weight adjustment unit 65 sets a 3 × 3 pixel local region centered on a sample point for which a weighting coefficient is obtained. Then, the weighting adjustment unit 65 obtains a weighting coefficient from the difference sum value of the luminance components for two frames before and after within the above-described local region. FIG. 21 is a diagram showing an example of a local region when obtaining a sample point C34 described later. In FIG. 21, the local region corresponding to the sample point C34 is indicated by a broken line.

  For example, the second color difference interpolation unit 64 may perform the calculations of Expressions (9) to (12) when obtaining the weighting coefficients of the sample points C33, C34, C43, and C44. The sample point C33 is a color difference sample point included in the nth frame thinned image.

  Here, in the description of the present specification, “eγ” indicates an inter-frame error in luminance at the sample point γ. “Yγ [n]” indicates the luminance component of the sample point γ of the nth frame. “MIN (δ)” is a function that returns the minimum value included in (δ). “Cγout” indicates the output value of the color difference after the weighted addition at the sample point γ.

  Step # 705: The adder 68 adds the color difference component of the first multiplier 66 weighted with the coefficient α and the color difference component of the second multiplier 67 weighted with the coefficient 1−α for each sample point. For example, for the sample points C33, C34, C43, and C44, the adding unit 68 of # 705 may obtain C33out, C34out, C43out, and C44out by the above-described equations (9) to (12). Then, the adding unit 68 outputs an image in YUV 4: 4: 4 format subjected to color difference interpolation to the outside. An example of the color difference channel in the output image of the adder 68 is shown in FIG. This is the end of the description of FIG.

(Operational effects of the first embodiment)
Hereinafter, the operational effects of the first embodiment will be described. The electronic camera 100 according to the first embodiment has a YUV4: 2: 0 format first thinned image and a second thinned image having different color difference thinning patterns from a YUV4: 4: 4 format YUV image when compressing a moving image. And generate Then, the electronic camera 100 encodes the first thinned image and the second thinned image described above in time series. Since the compressed moving image information after encoding is in the YUV 4: 2: 0 format, the amount of information can be greatly reduced as compared with the case of encoding without performing color difference thinning.

  In addition, when decoding the above-described compressed moving image information, the electronic camera 100 according to the first embodiment performs color difference interpolation processing using information on color difference components of images having different color difference thinning patterns, and YUV 4: 4: 4. Restore color image in format. Accordingly, it is possible to suppress color bleeding at the edge portion of the decoded image that occurs during compression with color difference thinning without using YUV 4: 4: 4 format image coding. Note that the color difference thinning pattern of the image can be explicitly notified to the video decoding device, for example, by adding accompanying information representing the color difference thinning pattern for each frame to the user data area. .

  In addition, since the compressed moving image information generated in the first embodiment conforms to the YUV 4: 2: 0 format, even in an apparatus that does not support the above-described color difference interpolation processing, a moving image in the YUV 4: 2: 0 format It can be decoded as an image as it is.

  Also, in the first embodiment, when the color difference thinning pattern between the encoding target frame and the reference frame is different in the motion compensation prediction, the motion efficiency prediction is performed by correcting the motion vector of the color difference to improve the encoding efficiency. be able to. At this time, the moving image compression apparatus gives correction information indicating the presence or absence of correction of the above-described color difference motion vector, whereby the color difference motion vector can be appropriately corrected on the video decoding apparatus side.

<Description of Second Embodiment>
FIG. 23 is a diagram illustrating an apparatus configuration example according to the second embodiment. The device of the second embodiment is a computer that functions as the above-described moving image compression device and moving image decoding device.

  A computer 201 illustrated in FIG. 23 includes a data reading unit 202, a storage device 203, a CPU 204, a memory 205, an input / output I / F 206, and a bus 207. The data reading unit 202, the storage device 203, the CPU 204, the memory 205 and the input / output I / F 206 are connected to each other via a bus 207. Furthermore, an input device 208 (a keyboard, a pointing device, etc.) and a monitor 209 are connected to the computer 201 via an input / output I / F 206. The input / output I / F 206 receives various inputs from the input device 208 and outputs display data to the monitor 209.

  The data reading unit 202 is used when reading uncompressed moving image information, the above-described compressed moving image information, and a program from the outside. For example, the data reading unit 202 communicates with a reading device (such as an optical disk, a magnetic disk, or a magneto-optical disk reading device) that acquires data from a removable storage medium, or an external device in accordance with a known communication standard. A communication device to perform (USB interface, LAN module, wireless LAN module, etc.).

  The storage device 203 is a storage medium such as a hard disk or a non-volatile semiconductor memory. The storage device 203 stores the above-described program and various data necessary for executing the program. The storage device 203 can also store compressed moving image information read from the data reading unit 202.

  The CPU 204 is a processor that comprehensively controls each unit of the computer 201. The CPU 204 functions as the image processing unit 111, the compression processing unit 112, and the decoding processing unit 113 of the first embodiment by executing the program.

  The memory 205 temporarily stores various calculation results in the program. The memory 205 is, for example, a volatile SDRAM.

  Here, in the second embodiment, uncompressed moving image information is acquired from the data reading unit 202 or the storage device 203, and the CPU 204 performs the compression operation shown in FIG. Alternatively, the above-described compressed moving image information is acquired from the data reading unit 202 or the storage device 203, and the CPU 204 performs the decoding operation shown in FIG. In the second embodiment, substantially the same effect as that of the first embodiment can be obtained.

<Supplementary items of the embodiment>
(Supplement 1): In the above-described embodiment, an example of the color difference thinning is performed in which the sample points of the lower left and upper right color differences of 2 × 2 pixels are thinned out. However, the sample points of the upper left and lower right color differences of 2 × 2 pixels are used. It may be thinned out.

  (Supplement 2): In the above-described embodiment, an example of compression encoding based on the main profile of MPEG-2 video has been described. However, in the present invention, other compression coding in JPEG, AVC, other YUV 4: 2: 0 format may be applied, or non-compression coding may be applied.

  (Supplement 3): The addition of the auxiliary information indicating the pattern of color difference thinning of the image is not limited to the example of the above-described embodiment. For example, by predefining a frame operation rule indicating a color difference thinning pattern that can be determined from temporal reference information of MPEG-2 video, the image color difference thinning pattern is implicitly notified to the decoding processing apparatus. You can also.

  (Supplement 4): In the above-described embodiment, the example in which the first thinned image and the second thinned image are alternately generated has been described. However, in the present invention, the first thinned image and the second thinned image are not necessarily generated alternately.

  FIG. 24 is a diagram illustrating another example of the encoding format of the moving image. FIG. 24 illustrates an example in which the first thinned image is generated every time two frames of the second thinned image are generated. In the case of FIG. 24, the first thinned image is an I picture or P picture, and the second thinned image is a B picture.

  (Supplement 5): In the above-described embodiment, the example in which the interpolation unit 42 performs the color difference interpolation process using an image that is one frame older than the image to be encoded has been described. However, the interpolation unit 42 can set a reference image without being limited to the above example as long as the image has a different color difference thinning pattern. For example, the interpolation unit 42 may perform the color difference interpolation process using an image that is several frames past the image to be encoded or an image that is future than the image to be encoded.

  In addition, the interpolation unit 42 may perform color difference interpolation using a plurality of images that are adjacent to the encoding target image in the time axis direction and have different color difference thinning patterns (see FIG. 25).

  FIG. 25 is a diagram illustrating another example of the interpolation unit 42. The example of FIG. 25 further includes a color difference synthesis unit 69, a third color difference interpolation unit 70, a third multiplication unit 71, and a fourth multiplication unit 72 in addition to the units shown in FIG. The color difference synthesizing unit 69 functions in the same manner as the color difference synthesizing unit 63 except that a color difference component is supplemented from an image one frame in the future with respect to an image serving as a reference for color difference interpolation. In addition, the third color difference interpolation unit 70 interpolates the color difference components of the image output from the color difference synthesis unit 69, and outputs a YUV 4: 4: 4 format image.

  Also, the weighting adjustment unit 65 in FIG. 25 uses the weighting coefficients α / 2 and (1−α) / 2 when adding the color difference component of the first color difference interpolation unit 62 and the color difference component of the second color difference interpolation unit 64. Obtained for each sample point (however, 0 ≦ α ≦ 1). Also, the weighting adjustment unit 65 adds the weighting coefficients β / 2 and (1-β) / 2 for each sampling point when adding the color difference component of the first color difference interpolation unit 62 and the color difference component of the third color difference interpolation unit 70. (Where 0 ≦ β ≦ 1). The third multiplication unit 71 multiplies the color difference component of the first color difference interpolation unit 62 by a coefficient β / 2. The fourth multiplication unit 72 multiplies the color difference component of the third color difference interpolation unit 64 by a coefficient (1-β) / 2. The adder 68 adds the outputs from the first multiplier 66 to the fourth multiplier 72.

  (Supplement 6): In the present invention, for example, color difference interpolation is performed by applying the super-resolution processing by a plurality of frames disclosed in Japanese Patent Application Laid-Open No. 2008-109375, and the color difference thinned image is converted into the YUV 4: 4: 4 format. Also good. In this case, the color difference can be interpolated more effectively by using neighboring frames having different color difference sample point positions.

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

DESCRIPTION OF SYMBOLS 11 ... Filter process part, 12 ... 1st switch, 13 ... 1st thinning-out part, 14 ... 2nd thinning-out part, 15 ... 2nd switch, 16 ... Encoding part, 17 ... Compression control part, 41 ... Decoding part, 42 DESCRIPTION OF SYMBOLS ... Interpolation part 43 ... Decoding control part 100 ... Electronic camera 101 ... Imaging optical system 102 ... Image pick-up element 103 ... Signal processing part 104 ... Image processing engine 105 ... 1st memory 106 ... 2nd memory, DESCRIPTION OF SYMBOLS 107 ... Recording I / F, 108 ... Monitor, 109 ... Operation part, 110 ... Storage medium, 111 ... Image processing part, 112 ... Compression processing part, 113 ... Decoding processing part, 201 ... Computer, 202 ... Data reading part, 203 ... Storage device 204 ... CPU 205 ... Memory 206 ... Input / output I / F, 207 ... Bus, 208 ... Input device, 209 ... Monitor

Claims (1)

A first decimation unit that decimates color difference components in a first pattern from a color image in a first format and generates a color image in a second format different from the first format;
A second thinning unit that thins out the color difference component from the color image of the first format with a second pattern different from the first pattern, and generates a color image of the second format;
A selection unit that distributes a plurality of first-format color images input in time series to the first thinning unit or the second thinning unit;
An encoding unit that encodes a plurality of color images of the second format generated in the first decimation unit or the second decimation unit along the time series;
A moving image compression apparatus comprising:

Images