JP2019022225A - Image compression apparatus and program - Google Patents

Abstract

To suppress coding efficiency decreases and image quality degradation during compression.SOLUTION: A decimation part of an image compression device generates a decimation image by discretely extracting sample points along a row direction from an image of a compression source such that points between sample points in odd-numbered rows and sample points in even-numbered rows are displaced in a row direction. A first image forming section forms a first image by extracting sample points in odd-numbered rows from the decimation image. A second image forming section forms a second image by extracting sample points in even-numbered rows from the decimation image. A coding section codes the first image and the second image, respectively. An output section outputs coded information of the first image and the coded information of the second image such that these pieces of information are associated with added information indicating a system for decoding to the decimation image.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image compression apparatus and a program.

  Conventionally, it has been known that the amount of information can be efficiently reduced by thinning sample points when compressing and encoding an image. For example, attention is paid to the point that it is difficult to visually recognize a decrease in resolution in an oblique direction as compared with the vertical direction and the horizontal direction, and an image is thinned out in an oblique direction and encoded (for example, Patent Document 1). ).

JP 2005-522958 A

  However, when compressing an image that has been thinned in an oblique direction, the correlation with adjacent sample points may be reduced depending on the pattern, resulting in a reduction in coding efficiency and image quality degradation.

  An image compression apparatus as an example of the present invention includes a thinning unit, a first image generation unit, a second image generation unit, an encoding unit, and an output unit. The thinning unit discretely extracts sample points along the row direction from the compression source image so that the positions of the sample points of the odd-numbered rows and the sample points of the even-numbered rows are shifted in the row direction. Generate an image. The first image generation unit generates a first image by extracting the sample points in the odd-numbered rows from the thinned image. The second image generation unit generates a second image by extracting even-numbered sample points from the thinned image. The encoding unit encodes the first image and the second image, respectively. The output unit outputs the encoded information of the first image and the encoded information of the second image in association with incidental information indicating a decoding method for the thinned image.

  According to an example of the present invention, it is possible to suppress a decrease in encoding efficiency and image quality degradation during compression.

The figure which shows the structural example of the electronic camera of 1st Embodiment. The figure which shows the structural example of the compression process part of FIG. (A)-(e): The figure which shows the thinning-out example at the time of the compression by the image of YUV4: 4: 4 format (A)-(e): The figure which shows the thinning-out example at the time of the compression by the image of a YUV4: 2: 2 format (A)-(e): The figure which shows an example of the thinning | decimation at the time of compression with the image of a YUV4: 2: 0 format (A)-(e): The figure which shows another example of the thinning | decimation at the time of compression with the image of YUV4: 2: 0 format The figure which shows the structural example of the encoding part of FIG. The figure which shows the structural example of the decoding process part of FIG. (A)-(c): The figure which shows the example of interpolation at the time of decoding with the image of YUV4: 4: 4 format (A)-(c): The figure which shows the example of interpolation at the time of the decoding at the image of YUV4: 2: 2 format (A)-(c): The figure which shows an example of the interpolation at the time of decoding with the image of a YUV4: 2: 0 format (A)-(c): The figure which shows another example of the interpolation at the time of decoding with the image of a YUV4: 2: 0 format The figure which shows the structural example of the decoding part of FIG. Flow chart showing an example of moving image compression operation in the compression processing unit (A)-(c): The figure which shows the example of the encoding format of the moving image in 1st Embodiment. Flow chart showing an example of a moving image decoding operation in the decoding processing unit The figure which shows the example of the interpolation process in an interpolation part The figure which shows the structural example of the compression process part in 2nd Embodiment. The figure which shows the structural example of the decoding process part in 2nd Embodiment. Flow chart showing an example of a moving image compression operation in the compression processing unit of the second embodiment (A), (b): The figure which shows the example of the encoding format of the moving image in 2nd Embodiment. The figure which shows the example which carries out motion compensation prediction of the 2nd image of a certain frame with reference to the 1st image of the same frame FIG. 20 is a flowchart illustrating an example of the compression encoding process in # 310 of FIG. 23 is a flowchart showing an example of motion compensation prediction processing in # 404 of FIG. A flowchart showing an example of a moving image decoding operation in the decoding processing unit of the second embodiment. FIG. 25 is a flowchart showing an example of the decoding process in # 604 of FIG. 26 is a flowchart showing an example of motion compensation prediction processing in # 705 of FIG. The figure which shows the apparatus structural example in 3rd Embodiment.

<Description of First Embodiment>
FIG. 1 is a diagram illustrating a configuration example of an electronic camera according to the first embodiment which is an example of an imaging device, an image compression device, and an image decoding device. Note that the electronic camera 100 of the first embodiment has a moving image shooting function.

  The electronic camera 100 includes an imaging optical system 101, an imaging element 102, a signal processing unit 103, an image processing engine 104, a first memory 105 and a second memory 106, a recording I / F 107, a monitor 108, and an operation. Part 109. Here, the signal processing unit 103, the first memory 105, the second memory 106, the recording I / F 107, the monitor 108, and the operation unit 109 are each connected to the image processing engine 104. The operation unit 109 is a switch that receives a user operation (for example, an instruction for moving image shooting, an instruction for mode switching, or the like).

  The imaging optical system 101 includes a plurality of lenses including, for example, a zoom lens and a focus lens. For simplicity, the imaging optical system 101 is illustrated as a single lens in FIG.

  The image sensor 102 is a device that images (captures) an image of a subject formed by a light beam that has passed through the imaging optical system 101. Note that in the moving image shooting mode, which is one of the shooting modes of the electronic camera 100, the image sensor 102 performs shooting of a moving image with recording in a nonvolatile storage medium (110). The output of the image sensor 102 is connected to the image processing engine 104. The above-described imaging device 102 may be a progressive scanning solid-state imaging device (for example, a CCD) or an XY addressing solid-state imaging device (for example, a CMOS).

  Here, a plurality of light receiving elements (pixels) are arranged in a matrix on the light receiving surface of the image sensor 102. A plurality of types of color filters that transmit light of different color components are arranged in the pixels of the image sensor 102 according to a predetermined color arrangement. Therefore, each pixel of the image sensor 102 outputs an electrical signal corresponding to each color component by color separation in the color filter. For example, in the first embodiment, red (R), green (G), and blue (B) color filters are periodically arranged on the light receiving surface according to a known Bayer array. Thereby, the image sensor 102 can acquire a color image at the time of photographing.

  The signal processing unit 103 performs analog signal processing (correlated double sampling, black level correction, etc.), A / D conversion processing, and digital signal processing (defective pixel correction, etc.) on the image signal input from the image sensor 102. ) In order. Data of the image (RAW image) output from the signal processing unit 103 is input to the image processing engine 104.

  The image processing engine 104 is a processor that comprehensively controls the operation of the electronic camera 100. For example, the image processing engine 104 controls autofocus (AF) and automatic exposure (AE) using an image signal input from the image sensor 102.

  The image processing engine 104 includes an image processing unit 111, a compression processing unit 112, and a decoding processing unit 113.

  The image processing unit 111 performs, for example, color interpolation processing, gradation conversion processing, white balance adjustment processing, color conversion processing, and the like on the RAW image. By the color interpolation process described above, a RAW image in which each color is arranged in a mosaic pattern in a Bayer array is converted into an RGB image having RGB information at each pixel. In addition, the RGB image after color interpolation is converted into a color image (YUV image) in the YUV color space by the color conversion process described above.

  The compression processing unit 112 is an example of an image compression device, and compresses and encodes an image output from the image processing unit 111. A configuration example of the compression processing unit 112 will be described later.

  The decoding processing unit 113 is an example of an image decoding device, and decodes the image data compression-encoded by the compression processing unit 112. A configuration example of the decoding processing unit 113 will be described later.

  The functional blocks of the image processing unit 111, the compression processing unit 112, and the decoding processing unit 113 included in the image processing engine 104 can be realized by an arbitrary processor, memory, or other LSI in terms of hardware, and in terms of software This is realized by a program loaded in the memory.

  The first memory 105 temporarily stores image data in the pre-process and post-process of image processing. For example, the first memory 105 is an SDRAM that is a volatile storage medium. The second memory 106 stores a program executed by the image processing engine 104 and various data used by this program. For example, the second memory 106 is a nonvolatile memory such as a flash memory.

  The recording I / F 107 has a connector for connecting the nonvolatile storage medium 110. The recording I / F 107 writes / reads moving image data to / from the storage medium 110 connected to the connector. The storage medium 110 is a hard disk, a memory card incorporating a semiconductor memory, or the like. In FIG. 1, a memory card is illustrated as an example of the storage medium 110.

  The monitor 108 is a display device that displays various images. For example, the monitor 108 controls the image processing engine 104 to display a viewfinder at the time of moving image shooting and to reproduce and display a captured moving image.

  Next, a configuration example of the compression processing unit 112 in FIG. 1 will be described with reference to FIG. In FIG. 2, each unit of the compression processing unit 112 is shown as a functional block realized by cooperation of hardware and software. In the first embodiment, a case where the compression source image is a moving image of a YUV image will be described. In the case of a moving image, each frame of the YUV image is an object of compression processing.

  The compression processing unit 112 in the first embodiment shown in FIG. 2 includes a color component separation unit 11, a first thinning unit 12, a second thinning unit 13, a first separation unit 14, a second separation unit 15, and a first image generation. Section 16, second image generation section 17, first encoding section 18, second encoding section 19, compression control section 20, and multiplexing section 21.

  The color component separation unit 11 separates an input YUV image frame into a luminance component (Y) channel and a color difference component (U, V) channel. The output of the color component separation unit 11 is connected to the first thinning unit 12 and the second thinning unit 13, respectively. The color component separation unit 11 outputs the Y channel (Y plane) of the YUV image to the first thinning unit 12, and outputs the U channel (U plane) and the V channel (V plane) of the YUV image to the second thinning unit 13. To do.

  If necessary, the color component separation unit 11 may perform a two-dimensional low-pass filter process on the output of each channel of the YUV image. Thereby, for example, occurrence of aliasing distortion due to thinning described later is suppressed. The low-pass filter is selected in accordance with the thinning characteristics described later, and a diamond-type low-pass filter is applied as an example.

  The first thinning unit 12 is an example of a thinning unit, and extracts a sample point from the Y channel of a YUV image to generate a luminance thinned image. The first decimation unit 12 discretely samples the sample points along the row direction so that the positions of the sample points of the odd-numbered rows and the sample points of the even-numbered rows in the Y channel are shifted in the row direction (X direction). To extract. Note that the output of the first thinning unit 12 is connected to the first separation unit 14.

  As an example, decimation when the input image is in YUV4: 4: 4 format, YUV4: 2: 2 format, and YUV4: 2: 0 format will be described. In the following description, the luminance sample point may be indicated by a circle (◯) in the figure, and the color difference sample point may be indicated by a cross (x) in the figure.

  FIG. 3 is a diagram showing an example of thinning out an image in the YUV 4: 4: 4 format. FIG. 3A shows an example of sample points before thinning, and FIG. 3B shows an example of sample points in a luminance thinned image. In FIG. 3A, the sample point of luminance (Y) and the sample point of color difference (U, V) are shown superimposed. In addition, in the YUV 4: 4: 4 format image shown in FIG. 3A, there is no thinning out of the sample points of luminance (Y) and color difference (U, V).

  When the input image is in YUV 4: 4: 4 format, the first thinning unit 12 discretely extracts sample points every other pixel in each row of the Y channel. At this time, the first thinning unit 12 extracts the sample points so that the odd-numbered sample points and the even-numbered sample points are shifted by one pixel in the row direction (X direction). Therefore, in the luminance thinned image shown in FIG. 3B, the luminance sample points are arranged in a checkered pattern with respect to the image before thinning.

  FIG. 4 is a diagram illustrating an example of thinning in an YUV 4: 2: 2 format image. FIG. 4A shows an example of sample points before thinning, and FIG. 4B shows an example of sample points in a luminance thinned image. In FIG. 4A, the sample point of luminance (Y) and the sample point of color difference (U, V) are shown superimposed. Further, in the YUV 4: 2: 2 format image shown in FIG. 4A, the sample points of the color difference (U, V) are thinned out by 1/2 only in the row direction (X direction).

  Even when the input image is in the YUV 4: 2: 2 format, the first thinning unit 12 extracts the Y channel sample points in the same manner as in the YUV 4: 4: 4 format. Therefore, in the luminance thinned image shown in FIG. 4B, the arrangement of the luminance sample points is the same as in FIG.

  FIG. 5 is a diagram illustrating an example of thinning in an YUV 4: 2: 0 format image. FIG. 5 (a) shows an example of sample points before thinning, and FIG. 5 (b) shows an example of sample points in a luminance thinned image. FIG. 6 is a diagram showing another example of thinning in an YUV 4: 2: 0 format image. FIG. 6A shows an example of sample points before thinning, and FIG. 6B shows an example of sample points in a luminance thinned image.

  In FIGS. 5A and 6A, the luminance (Y) sampling point and the color difference (U, V) sampling point are superimposed. Also, in the YUV 4: 2: 0 format images shown in FIGS. 5A and 6A, the sample points of the color difference (U, V) are respectively in the row direction (X direction) and the column direction (Y direction). 1/2 is thinned out. In the case of FIG. 5, the sample point of the color difference (U, V) is at a position where it overlaps the sample point of the luminance (Y). On the other hand, in the case of FIG. 6, the sample point of the color difference (U, V) is shifted by 0.5 pixels in the column direction (Y direction), and the sample point of the color difference (U, V) has the luminance (Y). The sample point does not overlap. Note that the color difference (U, V) sample points shown in FIG. 6A may be generated by, for example, averaging the color differences of two upper and lower sample points from a YUV 4: 4: 4 format image.

  Even when the input image is in the YUV 4: 2: 0 format, the first thinning unit 12 extracts the Y channel sample points in the same manner as in the YUV 4: 4: 4 format. Therefore, in the luminance thinned images shown in FIGS. 5B and 6B, the arrangement of the luminance sample points is the same as in FIG. 3B.

  The second thinning unit 13 is an example of a thinning unit, and generates a color difference thinned image by extracting sample points from the U channel and V channel of the YUV image. The second thinning unit 13 performs sampling points along the row direction so that the positions of the odd-numbered row sampling points and the even-numbered row sampling points in the U channel and the V channel are shifted in the row direction (X direction). Are discretely extracted. The above-described color difference thinned images are generated in the U channel and the V channel, respectively. The U-channel color difference thinned image and the V-channel color difference thinned image have the same sampling point extraction position. Note that the output of the second thinning unit 13 is connected to the second separation unit 15.

  Here, FIG. 3C shows an example of sample points in the color difference thinned image corresponding to the input image of FIG. When the input image is in the YUV 4: 4: 4 format, the second thinning unit 13 discretely extracts sample points every other pixel in each row of the U and V channels. At this time, the second thinning unit 13 extracts the sample points so that the odd-numbered sample points and the even-numbered sample points are shifted by one pixel in the row direction (X direction). Therefore, in the color difference thinning image shown in FIG. 3C, the color difference sample points are arranged in a checkered pattern. When the input image is in the YUV 4: 4: 4 format, the positions of the sample points of the luminance thinned image (FIG. 3B) and the color difference thinned image (FIG. 3C) are the same.

  FIG. 4C shows an example of sample points in the color difference thinned image corresponding to the input image of FIG. When the input image is in the YUV 4: 2: 2 format, the second thinning unit 13 discretely extracts every other sample point in each row of the U and V channels. At this time, the second thinning unit 13 extracts the sample points so that the odd-numbered sample points and the even-numbered sample points are shifted one by one in the row direction (X direction).

  In an input image of the YUV 4: 2: 2 format, the color difference sample points are thinned out in half in the row direction (X direction). Therefore, in the color difference thinned image shown in FIG. 3C, color difference sample points are extracted every three pixels in the row direction (X direction) with respect to the image before thinning. In the color difference thinning image shown in FIG. 3C, the color difference sample points are shifted by two pixels between the odd and even lines.

  FIG. 5C shows an example of sample points in the color difference thinned image corresponding to the input image of FIG. FIG. 6C shows an example of sample points in the color difference thinned image corresponding to the input image of FIG. When the input image is in the YUV 4: 2: 0 format, the second thinning unit 13 discretely extracts every other sample point in each row having the color difference sample points in the U and V channels. At this time, the second thinning unit 13 extracts the sample points so that the sample points in the odd-numbered rows and the sample points in the even-numbered rows are shifted one by one in the row direction (X direction).

  In the input image in the YUV 4: 2: 0 format, the color difference sample points are thinned out in half in the row direction (X direction) and in the column direction (Y direction). Therefore, in the color difference thinning image shown in FIG. 5C, every other row having color difference sample points is arranged. In the color difference thinned image shown in FIG. 5C, color difference sample points are extracted every three pixels in the row direction (X direction) with respect to the image before thinning. In the color difference thinning image shown in FIG. 5C, the color difference sample points are shifted by two pixels between the odd-numbered rows and the even-numbered rows.

  Note that the color difference thinned image in FIG. 6C is the same as the color difference thinned image in FIG. 5C except that the color difference sample points are shifted by 0.5 pixels in the column direction (Y direction). The pattern is common. Therefore, the overlapping description regarding FIG.

  The first separation unit 14 is connected to the first image generation unit 16 and the second image generation unit 17, respectively. The first separation unit 14 outputs information on odd-numbered rows in the luminance thinned image to the first image generation unit 16, and outputs information on even-numbered rows in the luminance thinned image to the second image generation unit 17. Output.

  The second separation unit 15 is connected to the first image generation unit 16 and the second image generation unit 17, respectively. The second separation unit 15 outputs information on odd-numbered rows in the color difference thinned image to the first image generation unit 16, and outputs information on even-numbered rows in the color difference thinned image to the second image generation unit 17. Output.

  The first image generation unit 16 generates a first image using information on odd-numbered rows respectively extracted from the luminance thinned image and the color difference thinned image. The first image generation unit 16 is connected to the first encoding unit 18.

  The second image generation unit 17 generates a second image using information on even-numbered rows respectively extracted from the luminance thinned image and the color difference thinned image. The second image generation unit 17 is connected to the second encoding unit 19.

  Here, examples of the first image and the second image will be described. The first image and the second image have the same data format as the compression source input image, but the number of sample points is 1/4. In addition, since the first image and the second image are generated by extracting different rows of the thinned image, the patterns of the sample points are different.

  FIG. 3D shows a first image in the YUV 4: 4: 4 format corresponding to FIG. FIG. 3E shows a second image in the YUV 4: 4: 4 format corresponding to FIG. In the case of FIG. 3, the positions of the luminance and color difference sample points are the same in both the first image and the second image. Further, between the first image and the second image, the position of the sample point is shifted by one pixel in the row direction (X direction) and the column direction (Y direction).

  FIG. 4D shows a first image in the YUV 4: 2: 2 format corresponding to FIG. FIG. 4E shows a second image in the YUV 4: 2: 2 format corresponding to FIG. In the case of FIG. 4, between the first image and the second image, the luminance sample points are shifted by one pixel in the row direction (X direction) and the column direction (Y direction). Further, between the first image and the second image, the sample point of the color difference is shifted by 2 pixels in the row direction (X direction) and 1 pixel in the column direction (Y direction).

  FIG. 5D shows a first image in the YUV 4: 2: 0 format corresponding to FIG. FIG. 5E shows a second image in the YUV 4: 2: 0 format corresponding to FIG. In the case of FIG. 5, between the first image and the second image, the luminance sample points are shifted by one pixel in the row direction (X direction) and the column direction (Y direction). Further, between the first image and the second image, the color difference sample points are shifted by two pixels in the row direction (X direction) and by two pixels in the column direction (Y direction).

  6D shows a first image in the YUV 4: 2: 0 format corresponding to FIG. 6A, and FIG. 6E shows YUV 4: 2: 0 corresponding to FIG. 6A. A second image of the format is shown. In the case of FIG. 6, between the first image and the second image, the luminance sample points are shifted by one pixel in the row direction (X direction) and the column direction (Y direction). Further, between the first image and the second image, the color difference sample points are shifted by two pixels in the row direction (X direction) and by two pixels in the column direction (Y direction). Note that the first image in FIG. 6D and the second image in FIG. 6E are different in that the sample point of the color difference is shifted by 0.5 pixels in the column direction (Y direction). It differs from d) and (e).

  The first encoding unit 18 is an example of an encoding unit, and compresses and encodes an input first image. For example, the first encoding unit 18 divides each frame of the moving image into blocks of a predetermined size, and compresses and encodes the moving image data using motion compensation prediction. Note that the output of the first encoding unit 18 is connected to the multiplexing unit 21. In the following description, information obtained by encoding the first image is also referred to as first encoded information.

  The second encoding unit 19 is an example of an encoding unit, and compresses and encodes an input second image. The configuration of the second encoding unit 19 is the same as that of the first encoding unit 18. Note that the output of the second encoding unit 19 is connected to the multiplexing unit 21. In the following description, information obtained by encoding the second image is also referred to as second encoded information.

  The compression control unit 20 comprehensively controls the operation of each unit of the compression processing unit 112 when compressing a moving image. In FIG. 2, illustration of the signal line connecting the compression control unit 20 and each unit is omitted as appropriate, except for the multiplexing unit 21.

  In addition, the compression control unit 20 generates incidental information including a decoding method from the first image and the second image to the thinned image. For example, the auxiliary information includes information on the data format of the compression source image, the thinning pattern of sample points in the luminance thinned image and the color difference thinned image, and the extraction pattern of the sample points in the first image and the second image.

  The multiplexing unit 21 is an example of an output unit, and multiplexes the first encoded information and the second encoded information in units of sub-pictures and outputs them in units of pictures. In addition, the multiplexing unit 21 adds the above-described incidental information to, for example, a header area of an encoded image. The multiplexing unit 21 may add the above-described supplementary information in units of moving image sequences. In the following description, the compressed moving image data generated by the compression processing unit 112 may be referred to as compressed moving image information.

  Next, a configuration example of the encoding unit (the first encoding unit 18 and the second encoding unit 19) in FIG. 2 will be described with reference to FIG. In FIG. 7, each part of an encoding part is shown by the functional block implement | achieved by cooperation of hardware or software.

  Here, as an image compression encoding method for performing motion compensation prediction, for example, MPEG-2 video defined in ISO / IEC13818-2 is known. In the above-described image compression coding system, compression coding is performed within a frame or between frames using I pictures, P pictures, and B pictures. An I picture is an intra-picture encoded image obtained by encoding that is completed only within the picture. A P picture is an inter-screen predictive encoded image obtained by predictive encoding using a maximum of one reference image. A B picture is an inter-screen bi-predictive encoded image obtained by predictive encoding using a maximum of two reference images.

  The encoding unit (18, 19) includes a first accumulation unit 31, a subtraction unit 32, an orthogonal transformation unit 33, a quantization unit 34, a variable length coding unit 35, an inverse quantization unit 36, an inverse orthogonal transformation unit 37, an addition Unit 38, second storage unit 39, prediction information generation unit 40, and prediction unit 41.

  The first accumulation unit 31 accumulates input images. For example, in the case of the first encoding unit 18, the first storage unit 31 stores the first image, and in the case of the second encoding unit 19, the first storage unit 31 stores the second image. The image accumulated in the first accumulation unit 31 is output to the subtraction unit 32 in the order of input as an image to be encoded. Note that the encoded image is sequentially deleted from the first storage unit 31.

  When generating a P picture or a B picture, the subtraction unit 32 outputs a difference signal (prediction error value) between the input original image and a prediction value (described later) generated by the prediction unit 41. Further, when generating the I picture, the subtracting unit 32 outputs the input original image signal as it is.

  When generating the I picture, the orthogonal transform unit 33 performs orthogonal transform on the signal of the original image input through the subtraction unit 32. Further, when generating the P picture or the B picture, the orthogonal transform unit 33 performs orthogonal transform on the above-described difference signal.

  The quantization unit 34 converts the frequency coefficient (orthogonal transform coefficient) in units of blocks input from the orthogonal transform unit 33 into a quantized coefficient. The output of the quantization unit 34 is input to the variable length encoding unit 35 and the inverse quantization unit 36, respectively.

  The variable length coding unit 35 performs variable length coding on prediction information such as a quantization coefficient and a motion vector, and outputs a coded bit stream of a moving image to the outside.

  The inverse quantization unit 36 inversely quantizes the quantization coefficient in units of blocks and decodes the frequency coefficient. The inverse orthogonal transform unit 37 performs inverse orthogonal transform on the frequency coefficient decoded by the inverse quantization unit 36 to decode a prediction error value (or an original image signal).

  The adding unit 38 adds the decoded prediction error value and a prediction value described later generated by the prediction unit 41. The decoded value (reference image) of the picture output from the adding unit 38 is stored in the second storage unit 39. Note that images that are not referred to in the subsequent motion compensation prediction are sequentially deleted from the second storage unit 39.

  The prediction information generation unit 40 uses the reference image of the second storage unit 39 to generate prediction information (such as a motion vector) for predicting the encoding target image. The prediction information is output to the prediction unit 41 and the variable length coding unit 35.

  The prediction unit 41 outputs a prediction value obtained by predicting the image to be encoded in units of blocks based on the prediction information and the reference image. The predicted value is output to the subtraction unit 32 and the addition unit 38.

  When it is determined that the prediction error due to motion compensation prediction is greater than a predetermined condition for a certain block, the prediction unit 41 uses the pixels in the block as a so-called intra macroblock without performing motion compensation prediction. Turn into.

  Next, a configuration example of the decoding processing unit 113 in FIG. 1 will be described with reference to FIG. In FIG. 8, each unit of the decoding processing unit 113 is shown as a functional block realized by cooperation of hardware and software.

  The decoding processor 113 includes a demultiplexer 51, a first decoder 52, a second decoder 53, a first separator 54, a second separator 55, a first combiner 56, a second combiner 57, a first An interpolation unit 58, a second interpolation unit 59, an image output unit 60, and a decoding control unit 61 are provided. Note that the above-described compressed moving image information is input to the decoding processing unit 113.

  The demultiplexing unit 51 is connected to the first decoding unit 52, the second decoding unit 53, and the decoding control unit 61. The multiplexing / separating unit 51 separates the first encoded information, the second encoded information, and the incidental information from the compressed video information. The separated first encoded information is output to the first decoding unit 52, and the separated second encoded information is output to the second decoding unit 53. The separated incidental information is output to the decoding control unit 61.

  The first decoding unit 52 is an example of a decoding unit, and generates a first image (FIG. 3 (d), FIG. 4 (d), FIG. 5 (d), and FIG. 6 (d)) from the first encoded information. Decrypt. The output of the first decoding unit 52 is connected to the first separation unit 54.

  The 2nd decoding part 53 is an example of a decoding part, Comprising: A 2nd image (FIG.3 (e), FIG.4 (e), FIG.5 (e), FIG.6 (e)) is shown from 2nd encoding information. Decrypt. The output of the second decoding unit 53 is connected to the second separation unit 55. The configuration of the second decoding unit 53 is the same as that of the first decoding unit 52.

  The first separation unit 54 is connected to the first synthesis unit 56 and the second synthesis unit 57. The first separation unit 54 separates the input first image into a Y channel and a U, V channel. The Y channel of the first image is output to the first combining unit 56, and the U and V channels of the first image are output to the second combining unit 57.

  The second separation unit 55 is connected to the first synthesis unit 56 and the second synthesis unit 57. The second separation unit 55 separates the input second image into the Y channel and the U and V channels. The Y channel of the second image is output to the first combining unit 56, and the U and V channels of the second image are output to the second combining unit 57.

  The first synthesizing unit 56 is an example of a synthesizing unit, and synthesizes the Y channel of the first image and the Y channel of the second image using the accompanying information. Thereby, the first synthesis unit 56 restores the luminance thinned image (FIG. 3B, FIG. 4B, FIG. 5B, FIG. 6B). Note that the output of the first synthesis unit 56 is connected to the first interpolation unit 58.

  The second synthesizing unit 57 is an example of a synthesizing unit, and synthesizes the U channel of the first image and the U channel of the second image using incidental information. In addition, the second combining unit 57 combines the V channel of the first image and the V channel of the second image using the incidental information. Thereby, the second composition unit 57 restores the above-described color difference thinned image (FIG. 3C, FIG. 4C, FIG. 5C, FIG. 6C). The output of the second synthesis unit 57 is connected to the second interpolation unit 59.

  The first interpolation unit 58 is an example of an interpolation unit, and interpolates the sample points of the restored luminance thinned image within the frame. Accordingly, the first interpolation unit 58 converts the luminance thinned image (Y) into a luminance interpolation image (Y) having the same format as the Y channel of the compression source image. Note that the output of the first interpolation unit 58 is connected to the image output unit 60.

  The second interpolation unit 59 is an example of an interpolation unit, and interpolates the sample points of the restored U and V channel color difference thinned images. Thus, the second interpolation unit 59 converts the color difference thinned image (U, V) into a color difference interpolated image (U, V) having the same format as the U, V channels of the compression source image. Note that the output of the second interpolation unit 59 is connected to the image output unit 60.

  The image output unit 60 associates the luminance interpolation image and the color difference interpolation image of the same frame and outputs them as one decoded frame. The luminance interpolation image has the same image size as the Y channel of the compression source image, and the color difference interpolation image has the same image size as the U and V channels of the compression source image. Therefore, each YUV component of the decoded frame is restored to the same image size as the compression source image.

  Here, examples of the above-described luminance interpolation image, color difference interpolation image, and decoded frame will be described. In the drawing of each image at the time of decoding, the interpolated luminance sample point is indicated by a dotted circle, and the interpolated color difference sample point is indicated by a broken cross.

  FIG. 9A shows a YUV 4: 4: 4 format luminance interpolation image corresponding to FIG. FIG. 9B shows a color difference interpolation image in the YUV 4: 4: 4 format corresponding to FIG. FIG. 9 (c) shows a decoded frame using FIG. 9 (a) and FIG. 9 (b).

  FIG. 10 (a) shows a YUV 4: 2: 2 format luminance interpolation image corresponding to FIG. 4 (a). FIG. 10B shows a YUV 4: 2: 2 format color difference interpolation image corresponding to FIG. FIG. 10 (c) shows a decoded frame using FIG. 10 (a) and FIG. 10 (b).

  FIG. 11A shows a luminance-interpolated image in the YUV 4: 2: 0 format corresponding to FIG. FIG. 11B shows a YUV 4: 2: 0 format color difference interpolation image corresponding to FIG. FIG.11 (c) has shown the decoding frame using Fig.11 (a) and FIG.11 (b).

  FIG. 12A shows a YUV 4: 2: 0 format luminance interpolation image corresponding to FIG. FIG. 12B shows a color difference interpolation image in the YUV 4: 2: 0 format corresponding to FIG. FIG. 12 (c) shows a decoded frame using FIG. 12 (a) and FIG. 12 (b).

  The decoding control unit 61 comprehensively controls the operation of each unit of the decoding processing unit 113 when decoding a moving image. In FIG. 8, the illustration of the signal line connecting the decoding control unit 61 and each unit is omitted as appropriate, except for the demultiplexing unit 51.

  In addition, the decoding control unit 61 performs control for restoring the luminance thinned image and the color difference thinned image and control for generating the luminance interpolation image and the color difference interpolated image using the auxiliary information separated from the compressed moving image information.

  Next, a configuration example of the decoding unit (the first decoding unit 52 and the second decoding unit 53) in FIG. 8 will be described with reference to FIG. In FIG. 13, each unit of the decoding unit is shown as a functional block realized by cooperation of hardware and software. Note that the functional blocks of the decoding units 52 and 53 shown in FIG. 13 may be shared with the functional blocks of the same name of the encoding units 18 and 19 shown in FIG.

  The decoding unit (52, 53) includes a code decoding unit 71, an inverse quantization unit 72, an inverse orthogonal transform unit 73, an addition unit 74, a first accumulation unit 75, a second accumulation unit 76, and a prediction unit 77. .

  The code decoding unit 71 decodes a coded bit stream of an input moving image and outputs prediction information such as a quantization coefficient and a motion vector. The decoded quantization coefficient is input to the inverse quantization unit 72, and the decoded prediction information is input to the prediction unit 77.

  The inverse quantization unit 72 inversely quantizes the quantization coefficient in units of blocks and decodes the frequency coefficient. The inverse orthogonal transform unit 73 performs inverse orthogonal transform on the frequency coefficient decoded by the inverse quantization unit 72 to decode a prediction error value (or an original image signal).

  The addition unit 74 adds the decoded prediction error value and the prediction value generated by the prediction unit 77 to output a decoded image in units of blocks. The decoded value of the image output from the adding unit 74 is input to the first storage unit 75 and the second storage unit 76, respectively.

  The first accumulation unit 75 accumulates the decoded first image or second image until it is output to the subsequent circuit (the first separation unit 54 or the second separation unit 55). Note that images that have been output to the subsequent circuit are sequentially deleted from the first storage unit 75. The second storage unit 76 stores the decoded image value as a reference image. Note that images that are not referred to in the subsequent motion compensation prediction are sequentially deleted from the second accumulation unit 76.

  The prediction unit 77 outputs a prediction value obtained by predicting the decoding target image in units of blocks to the addition unit 74 based on the prediction information and the reference image.

(Example of video compression)
Hereinafter, an example of a moving image compression operation in the first embodiment will be described.

  When the electronic camera 100 according to the first embodiment receives a shooting instruction in the moving image shooting mode, the image processing engine 104 drives the image sensor 102 to start moving image shooting. A frame of a moving image output from the image sensor 102 is subjected to a series of signal processing and image processing by the signal processing unit 103 and the image processing unit 111, and then compression encoded by the compression processing unit 112. The compressed moving image information compressed by the compression processing unit 112 is recorded on the storage medium 110 connected to the recording I / F 107 under the control of the image processing engine 104.

  FIG. 14 is a flowchart showing an example of a moving image compression operation in the compression processing unit 112.

  Step # 101: The compression control unit 20 of the compression processing unit 112 performs setting of a thinning pattern of sample points in the luminance thinned image and the color difference thinned image, and setting of a composite pattern of the first image and the second image. The setting of # 101 is executed only once in the first frame of the moving image.

  Step # 102: The color component separation unit 11 performs low pass on the input YUV image frame (FIG. 3 (a), FIG. 4 (a), FIG. 5 (a), FIG. 6 (a)) as necessary. Apply filtering. Thereafter, the color component separation unit 11 separates the YUV image into a luminance component and a color difference component. The Y channel of the YUV image is output to the first thinning unit 12, and the U and V channels of the YUV image are output to the second thinning unit 13.

  Step # 103: The first decimation unit 12 extracts sample points from the Y channel and generates luminance decimation images (FIG. 3 (b), FIG. 4 (b), FIG. 5 (b), and FIG. 6 (b)). To do.

  Step # 104: The second thinning unit 13 extracts the sample points from the U and V channels and extracts the color difference thinned images (FIG. 3 (c), FIG. 4 (c), FIG. 5 (c), FIG. 6 (c)). Is generated. In addition, the process of # 103 and the process of # 104 may be performed in parallel.

  Step # 105: The first separation unit 14 outputs information on odd-numbered rows in the luminance thinned image (# 103) to the first image generation unit 16, and even-numbered in the luminance thinned image (# 103). Is output to the second image generation unit 17.

  Step # 106: The second separation unit 15 outputs the information on the odd-numbered rows in the color difference thinned image (# 104) to the first image generation unit 16, and the even number in the color difference thinned image (# 104). Is output to the second image generation unit 17. In addition, the process of # 105 and the process of # 106 may be performed in parallel.

  Step # 107: The first image generation unit 16 uses the information on the odd-numbered rows extracted from the luminance thinned image and the color difference thinned image, respectively, to display the first image (FIG. 3D, FIG. 4D, FIG. 5 (d), FIG. 6 (d)) is generated.

  Step # 108: The second image generation unit 17 uses the information of the even-numbered rows extracted from the luminance thinned image and the color difference thinned image, respectively, to generate the second image (FIG. 3 (e), FIG. 4 (e), FIG. 5 (e) and FIG. 6 (e)) are generated. In addition, the process of # 107 and the process of # 108 may be performed in parallel.

  Step # 109: The compression control unit 20 generates the accompanying information described above based on the setting of # 101. Note that the compression control unit 20 may encode the accompanying information.

  Step # 110: The first encoding unit 18 compresses and encodes the first image (# 107) in accordance with, for example, the main profile of MPEG-2 video. Thereby, the first encoded information is generated. Here, the first encoding unit 18 in the first embodiment performs motion compensation prediction between the first images of different frames, but does not perform motion compensation prediction between the first image and the second image. .

  Step # 111: The second encoding unit 19 compresses and encodes the second image (# 108) in accordance with, for example, the main profile of MPEG-2 video. Thereby, the second encoded information is generated. Here, the second encoding unit 19 in the first embodiment performs motion compensation prediction between second images of different frames, but does not perform motion compensation prediction between the first image and the second image. . In addition, the process of # 110 and the process of # 111 may be performed in parallel.

  Here, FIGS. 15A to 15C are diagrams illustrating examples of the encoding format of the moving image in the first embodiment. In FIG. 15, “In” indicates the nth I picture, “Pn” indicates the nth P picture, and “Bn” indicates the nth B picture. The first image is indicated by a subscript number “1”, and the second image is indicated by a subscript number “2”.

  FIG. 15A shows an example in which each frame of a moving image is encoded as an I picture. In the case of FIG. 15A, only intra prediction encoding is performed, and inter prediction encoding is not performed between different images.

  FIG. 15B shows an example in which a moving image is encoded with a combination of an I picture and a P picture. In encoding a P picture, inter-screen predictive encoding is performed independently between first images or between second images with reference to the immediately preceding (most recent past) I picture or P picture in the time axis direction.

  FIG. 15C shows an example in which a moving image is encoded with a combination of an I picture, a P picture, and a B picture. In the case of FIG. 15C, an I picture and a P picture are generated every two frames, and a B picture for two frames is inserted between the I picture and the P picture. In encoding a B picture, inter-picture bi-predictive encoding is performed independently between first images or between second images with reference to I and P pictures adjacent to each other in the time axis direction.

  When compressing a P picture, the encoding unit selects one reference image for each block from among reference image candidates, and performs motion compensation prediction of each block using a decoded value of the reference image. In addition, when compressing a B picture, the encoding unit selects one past and future reference images for each block from among reference image candidates, and uses the decoded values of the two reference images for each block. Perform motion compensation prediction.

  Step # 112: The multiplexing unit 21 multiplexes the first encoded information and the second encoded information in units of pictures, adds the above-mentioned incidental information, and outputs compressed moving picture information. Note that the compressed moving image information is recorded in the storage medium 110.

  Step # 113: The compression control unit 20 determines whether or not compression of the entire input moving image has been completed. When the above requirements are satisfied (YES side), the compression control unit 20 ends the compression operation. On the other hand, when the above requirement is not satisfied (NO side), the process returns to # 102 and the next frame is compressed. Above, description of FIG. 14 is complete | finished.

(Example of video decoding operation)
Hereinafter, an example of a moving image decoding operation according to the first embodiment will be described.

  When the electronic camera 100 according to the first embodiment receives an instruction to reproduce the compressed moving image information compressed by the compression processing unit 112, the image processing engine 104 reads the compressed moving image information from the storage medium 110 connected to the recording I / F 107. To get. Then, the decoding processing unit 113 decodes the compressed moving image information. The decoded moving image is reproduced and displayed, for example, on the monitor 108 under the control of the image processing engine 104.

  FIG. 16 is a flowchart illustrating an example of a moving image decoding operation in the decoding processing unit 113.

  Step # 201: The decoding processing unit 113 acquires compressed moving image information from the storage medium 110.

  Step # 202: The multiplexing / separating unit 51 separates information (first encoded information, second encoded information, and incidental information) for one frame of the compressed moving image information. The separated first encoded information is output to the first decoding unit 52, and the separated second encoded information is output to the second decoding unit 53. The separated incidental information is output to the decoding control unit 61.

  Step # 203: The decoding control unit 61 decodes the incidental information as necessary and refers to the incidental information. Thereby, the decoding control unit 61 acquires information on the thinning pattern of the sample points in the luminance thinned image and the color difference thinned image, and information on the combined pattern of the first image and the second image.

  Step # 204: The first decoding unit 52 converts the first image (FIG. 3 (d), FIG. 4 (d), FIG. 5 (d) from the first encoded information in accordance with, for example, the main profile of MPEG-2 video. , FIG. 6D) is decoded.

  Step # 205: The second decoding unit 53 obtains the second image (FIG. 3 (e), FIG. 4 (e), FIG. 5 (e) from the second encoded information in accordance with, for example, the main profile of MPEG-2 video. , FIG. 6 (e)) is decoded. Note that the process of # 204 and the process of # 205 may be executed in parallel.

  Step # 206: The first separation unit 54 separates the input first image (# 204) into the Y channel and the U and V channels. The Y channel of the first image is output to the first combining unit 56, and the U and V channels of the first image are output to the second combining unit 57.

  Step # 207: The second separation unit 55 separates the input second image (# 205) into the Y channel and the U and V channels. The Y channel of the second image is output to the first combining unit 56, and the U and V channels of the second image are output to the second combining unit 57. Note that the process of # 206 and the process of # 207 may be executed in parallel.

  Step # 208: The first combining unit 56 combines the Y channel of the first image and the Y channel of the second image using the accompanying information. Thereby, the first synthesis unit 56 restores the luminance thinned image (FIG. 3B, FIG. 4B, FIG. 5B, FIG. 6B).

  For example, the first combining unit 56 acquires information on odd-numbered rows of the Y channel from the first image, and acquires information on even-numbered rows of the Y channel from the second image. The first combining unit 56 may restore the luminance thinned image by rearranging the odd-numbered row information and the even-numbered row information so that they are alternately arranged in the column direction.

  Step # 209: The second combining unit 57 combines the U and V channels of the first image and the U and V channels of the second image using the accompanying information. As a result, the first combining unit 56 restores the U and V channel luminance thinned images (FIG. 3C, FIG. 4C, FIG. 5C, and FIG. 6C), respectively.

  For example, the second combining unit 57 acquires information on odd-numbered rows of the U and V channels from the first image, and acquires information on even-numbered rows of the U and V channels from the second image. Then, the second combining unit 57 may restore the color difference thinned image by rearranging the odd-numbered row information and the even-numbered row information so that they are alternately arranged in the column direction. Note that the process of # 208 and the process of # 209 may be executed in parallel.

  Step # 210: The first interpolation unit 58 inter-frame interpolates the sample points of the luminance thinned image restored in # 208. As a result, the first interpolation unit 58 converts the luminance thinned image of the Y channel into luminance interpolation images (FIG. 9A, FIG. 10A, FIG. 11A, and FIG. 12A).

  FIG. 17 is a diagram illustrating an example of interpolation processing in the interpolation unit. As an example, in the example of FIG. 17, a case is considered in which intra-frame interpolation is performed on sample points Y45 adjacent to sample points Y35, Y44, Y46, and Y55 in the luminance thinned image. At this time, the first interpolation unit 52 may obtain the luminance component of the sample point Y45 to be interpolated by the calculation of Expression (1).

  Here, in the description of the present specification, “Sγh” indicates the horizontal correlation at the sample point γ, and “Sγv” indicates the vertical correlation at the sample point γ. “Kγh” indicates a horizontal weighting value at the sample point γ, and “Kγv” indicates a vertical weighting value at the sample point γ.

  Step # 211: The second interpolation unit 59 inter-frame interpolates the sample points of the color difference thinned image restored in # 209. As a result, the second interpolation unit 59 converts the U and V channel color difference thinned images into color difference interpolated images (FIG. 9B, FIG. 10B, FIG. 11B, and FIG. 12B), respectively. To do. Interpolation processing in the second interpolation unit 59 is substantially the same as the processing in the first interpolation unit 58 (see FIG. 17), and thus a duplicate description is omitted. In addition, the process of # 210 and the process of # 211 may be performed in parallel.

  Step # 212: The image output unit 60 associates the luminance interpolated image and the color difference interpolated image of the same frame with one decoded frame (FIG. 9 (c), FIG. 10 (c), FIG. 11 (c), FIG. 12 (c)).

  Step # 213: The decoding control unit 61 determines whether or not the decoding of the entire compressed moving image information has been completed. If the above requirement is satisfied (YES side), the decoding control unit 61 ends the decoding operation. On the other hand, when the above requirement is not satisfied (NO side), the process returns to # 201 and the next frame is decoded. This is the end of the description of FIG.

(Operational effects of the first embodiment)
Hereinafter, the operational effects of the first embodiment will be described. In the electronic camera 100 according to the first embodiment, when compressing a moving image, the Y channel of the YUV image and the U channel are arranged so that the positions of the sample points in the odd-numbered rows and the sample points in the even-numbered rows are shifted in the row direction. Sample points are thinned out for the V channel and V channel, respectively. In addition, the electronic camera 100 generates the first image by extracting the sample points in the odd-numbered rows from the luminance thinned image and the color difference thinned image after the thinning. In addition, the electronic camera 100 extracts the sample points in the even-numbered rows from the luminance thinned image and the color difference thinned image after the thinning, and generates a second image. Then, the electronic camera 100 encodes the first image to generate first encoded information, and encodes the second image to generate second encoded information. Since the first image and the second image have 1/4 the number of sample points compared to the compression source image, the data amount of the image is smaller than that before thinning.

  Also, for example, when compression is performed in a state where the interval between images thinned in the diagonal direction is reduced, the correlation between adjacent sample points is reduced in an image including a vertical edge or a horizontal edge, and the compression Declining encoding efficiency and image quality may occur. On the other hand, in the first embodiment, since the image thinned out in the oblique direction is compressed separately into the first image and the second image, the reduction of the encoding efficiency and the deterioration of the image quality at the time of compression are suppressed as compared with the above case. it can.

  In addition, when decoding the compressed moving image information, the electronic camera 100 according to the first embodiment rearranges the first image and the second image to restore the luminance thinned image and the color difference thinned image. The electronic camera 100 then interpolates the luminance thinned image and the color difference thinned image, and outputs a decoded frame having the same format as the compression source image. Note that the image thinning pattern can be explicitly notified to the image decoding device by adding additional information to the compressed moving image information by the image compression device. Further, in the first embodiment, a thinned image is restored using two images having different sample point patterns, and a decoded frame is generated by interpolating the thinned image. Therefore, the first image and the second image are reproduced as they are. The image quality is higher than in the case where

  In addition, the compressed moving image information of the first embodiment can be used for the image quality of the first image or the second image in accordance with the main profile of MPEG-2 video even in a general decoding device that does not include the decoding processing unit 113 described above. Can be decrypted and played.

  In the compressed moving image information according to the first embodiment, the first encoded information and the second encoded information are compressed independently of each other, so that only the first encoded information or the second encoded information is used. The image can be decoded and reproduced with the image quality. For example, when transferring a moving image using a communication line, only one encoded information is transmitted on a low speed line, and compressed moving image information is transmitted on a high speed line.

<Description of Second Embodiment>
The second embodiment is a modification of the first embodiment, and only the configuration of the compression processing unit 112 and the decoding processing unit 113 is different. Therefore, in the following description, elements common to the first embodiment are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 18 is a diagram illustrating a configuration example of the compression processing unit 112 in the second embodiment. The compression processing unit 112 in the second embodiment includes a color component separation unit 11, a first thinning unit 12, a second thinning unit 13, a first separation unit 14, a second separation unit 15, a first image generation unit 16, A two-image generation unit 17, a compression control unit 20, and an encoding processing unit 22.

  Here, among the compression processing units 112 in the second embodiment, the color component separation unit 11, the first thinning unit 12, the second thinning unit 13, the first separation unit 14, the second separation unit 15, and the first image generation. The configurations and functions of the unit 16, the second image generation unit 17, and the compression control unit 20 are the same as those in the first embodiment. Note that the outputs of the first image generation unit 16 and the second image generation unit 17 in the second embodiment are connected to the encoding processing unit 22, respectively.

  The encoding processing unit 22 is an example of an encoding unit and an output unit, and has a function of compressing and encoding the input first image and second image and a function of adding incidental information. The encoding unit built in the encoding processing unit 22 is configured in the same manner as in FIG. Therefore, redundant description of each element of the encoding unit included in the encoding processing unit 22 is omitted. In addition, the encoding processing unit 22 in the second embodiment performs inter-screen bi-predictive encoding between the first image and the second image.

  FIG. 19 is a diagram illustrating a configuration example of the decryption processing unit 113 in the second embodiment. The decoding processing unit 113 in the second embodiment includes a decoding unit 62, a first separation unit 54, a second separation unit 55, a first synthesis unit 56, a second synthesis unit 57, a first interpolation unit 58, and a second interpolation unit. 59, an image output unit 60, and a decoding control unit 61.

  Here, in the decoding processing unit 113 in the second embodiment, the compressed moving image information is input to the decoding unit 62. The decoding unit 62 is configured in the same manner as in FIG. For this reason, redundant description of each element of the decoding unit 62 is omitted. In addition, the decoding unit 62 decodes each of the first image and the second image encoded by the inter-screen bi-predictive encoding. Then, the decoding unit 62 outputs the decoded first image to the first separation unit 54, and outputs the decoded second image to the second separation unit 55. Further, the decoding unit 62 outputs the incidental information to the decoding control unit 61.

  Of the decoding processing unit 113 in the second embodiment, the first separation unit 54, the second separation unit 55, the first synthesis unit 56, the second synthesis unit 57, the first interpolation unit 58, and the second interpolation unit 59. The configurations and functions of the image output unit 60 and the decoding control unit 61 are the same as those in the first embodiment.

(Example of video compression)
An example of a moving image compression operation in the second embodiment will be described below. FIG. 20 is a flowchart illustrating an example of a moving image compression operation in the compression processing unit 112 according to the second embodiment. Note that the processing from # 301 to # 308 shown in FIG. 20 corresponds to the processing from # 101 to # 109 shown in FIG.

  Step # 310: The encoding processing unit 22 compresses and encodes the first image (# 307) and the second image (# 308) in accordance with, for example, the main profile of MPEG-2 video. As a result, first encoded information and second encoded information are generated.

  FIGS. 21A and 21B are diagrams illustrating examples of moving image encoding formats according to the second embodiment. In FIG. 21, “In” indicates the nth I picture, “Pn” indicates the nth P picture, and “Bn” indicates the nth B picture. The first image is indicated by a subscript number “1”, and the second image is indicated by a subscript number “2”.

  FIG. 21A shows an example in which a moving image is encoded by a combination of an I picture and a P picture. In the case of FIG. 21A, inter-screen prediction encoding is performed with reference to the past first image and second image in the time axis direction.

  In the case of FIG. 21A, in the motion compensation prediction of the first image that is a P picture, the first image (I picture / P picture) or the second image (P picture) that is one frame past each is the reference image candidate. Become. In addition, in the motion compensation prediction of the second image to be a P picture, the second image (P picture) that is one frame past or the first image (I picture / P picture) separated from the same frame is a candidate for the reference image. .

  FIG. 21B shows an example in which a moving image is encoded with a combination of an I picture, a P picture, and a B picture. In the case of FIG. 21B, in the first image, an I picture and a P picture are generated every two frames, and a B picture for two frames is inserted between the I picture and the P picture. In the second image, only P pictures are generated every two frames, and B pictures for two frames are inserted between the P pictures.

  In the case of FIG. 21B, in the motion compensation prediction of the first image and the second image that are P pictures, the first picture (I picture / P picture) or the second picture corresponding to the frame of the immediately preceding I picture or P picture is used. An image (P picture) is a candidate for a reference image.

  In the case of FIG. 21B, in motion compensated prediction of the first image and the second image that are B pictures, (1) the first image or the first image separated from the frame of the most recent I picture or P picture on the past side. One of the two images and (2) one of the first image and the second image separated from the frame of the latest I picture or P picture on the future side are candidates for the reference image.

  By the way, the positions of the sample points of the pixels at the same position in the first image and the second image are different in the original image.

  Due to the difference in the position of the sample points, relatively small inter-frame prediction errors are caused in the still part of the image by inter-frame prediction without motion compensation between the first images or the second images of different frames. Can be encoded.

  On the other hand, in the motion portion of an image, for example, when motion compensation prediction is performed on a first image of a certain frame, a second image of a different frame is referred to rather than referring to the first image of a different frame. When the prediction error can be made smaller, or when the second image of a certain frame is subjected to motion compensation prediction, the motion compensation prediction can be made smaller from the first image of the same frame than the second image of a different frame. There is. The former case is a case where the moving part of the first image to be encoded has moved from the pixel position of the second image in a different frame or, for example, moved to the pixel position of the second image in a different frame. The latter case is a case where the prediction error cannot be reduced by motion compensated prediction referring to the second image of a different frame due to a change in the pattern of the second image to be encoded or occlusion. Therefore, as shown in FIG. 21, by selecting the reference images of the first image and the second image in units of blocks and performing motion compensation prediction, the coding efficiency can be greatly improved.

  In addition, when motion compensation prediction is performed with reference to the other image of the same frame, a difference in sample point position may be compensated. In the following description, the arrangement of sample points (pixels P) is indicated by circles on the image in the drawings. Further, the positional deviation of the reference pixel of the prediction source image with respect to the position of the target pixel of the prediction destination image is referred to as a motion vector MV [x, y]. In the motion vector MV, x increases due to a right shift, and x decreases due to a left shift. Similarly, in the motion vector MV, y increases due to a downward shift, and y decreases due to an upward shift.

  FIG. 22 is an example of motion compensation prediction of a second image of a certain frame with reference to the first image of the same frame. FIG. 22 illustrates an example in which the pixel value at the target pixel P2x of the prediction destination image (second image) is obtained as an average value of four pixels (P1a to P1d) of the prediction source image (first image) adjacent to the target pixel. Show. In the case of FIG. 22, since the pixel value of the target pixel is predicted by the average of four adjacent pixels, the coding distortion included in the decoded value of the prediction source image can be suppressed.

  In the example of FIG. 22, the P1 pixel of the first image is arranged at the upper left of the color arrangement, and the P2 pixel of the second image is arranged at the lower right of the color arrangement. In the case of FIG. 22, the pixel P1a of the first image and the pixel P2x of the second image are encoded as sample points at the same position. Therefore, in the example of FIG. 22, the motion vector MV is [0.5, 0.5].

  In the motion compensated prediction of FIG. 22 described above, there is a high probability that a block with a relatively small change in pattern is selected. Since a moving image includes many portions having no motion, the frequency of selecting the prediction of FIG. 22 is high in the compression encoding of the first image and the second image. Therefore, the encoding processing unit 22 may assign the shortest code in the case of motion compensated prediction in FIG. 22 to increase the compression efficiency.

For example, when motion compensation prediction is performed on a second image of a certain frame with reference to the first image of the same frame, if correction is performed by uniformly subtracting [0.5, 0.5] from the motion vector, MPEG-2 Since [0, 0] is the shortest code allocation for video or the like, the compression efficiency can be increased. In addition, when decoding the corrected data, [0.5, 0.5] may be added and corrected at the time of decoding.

  In addition, as described with reference to FIGS. 4, 5, and 6, the arrangement of the color difference sample points with respect to the luminance sample points is different between the first image and the second image according to the data format of the compression source image. May be different. Therefore, the encoding processing unit 22 needs to obtain the correction amount of the motion vector applied in the U and V channels according to the data format of the compression source image.

  As an example, when performing U-channel and V-channel motion compensation prediction in the first and second images in the YUV 4: 2: 2 format (FIGS. 4D and 4E), the first and second images are used. Thus, the sample point of the color difference is shifted by 0.5 pixel in the horizontal direction in terms of Y channel. In addition, when performing U- and V-channel motion compensation prediction in the first and second images in the YUV 4: 2: 0 format (FIGS. 5D and 5E), the first image and the second image are used. The sample point of the color difference is shifted by 0.5 pixels in the horizontal and vertical directions in terms of Y channel. In general, motion compensation prediction for U and V channels uses a motion vector obtained by scaling a Y channel motion vector according to a data format. If the above-mentioned shift is not taken into consideration, the U and V channels are motion-compensated and predicted with the shifted motion vector, so that the coding efficiency is lowered. In the motion compensation prediction of the U and V channels, the coding efficiency can be improved by correcting the above-described shift (that is, correcting the motion vector) according to the data format.

  Note that the correction of the motion vector in the motion compensated prediction between the first image and the second image may be standardized by a compression method in advance, and correction information indicating whether or not the above-described correction of the motion vector is necessary is provided at the time of encoding. By doing so, it may be clearly shown to the image decoding apparatus.

  Step # 311: The encoding processing unit 22 outputs compressed moving image information obtained by multiplexing the first encoded information, the second encoded information, and the above-described incidental information in units of pictures. Note that the compressed moving image information is recorded in the storage medium 110.

  Here, when motion compensation prediction is performed on the first image and the second image in # 310, the compression control unit 20 generates correction information indicating whether or not the above-described motion vector needs to be corrected. Then, the encoding processing unit 22 in # 311 further adds the above-described correction information to the compressed moving image information. For example, the encoding processing unit 22 can include the above-described supplementary information and correction information in the user data area following the picture coding extension when compression encoding is performed in accordance with the main profile of MPEG-2 video.

  Step # 312: The compression control unit 20 determines whether or not compression of the entire input moving image has been completed. When the above requirements are satisfied (YES side), the compression control unit 20 ends the compression operation. On the other hand, when the above requirement is not satisfied (NO side), the process returns to # 302 and the next frame is compressed. Above, description of FIG. 20 is complete | finished.

  FIG. 23 is a flowchart showing an example of the compression encoding process in # 310 of FIG.

  Step # 401: The compression control unit 20 identifies the type of image (first image, second image) to be encoded. In addition, the compression control unit 20 determines the encoding format (I picture, P picture, B picture) of an image to be encoded.

  Step # 402: The compression control unit 20 determines whether or not it is necessary to correct a motion vector in motion compensation prediction at the time of compression according to a combination of an image to be encoded and a reference image.

  For example, when inter-screen predictive coding is performed between the first images and between the second images, the positions of the sample points match between the image to be coded and the reference image. However, when inter-picture predictive encoding is performed between the first image and the second image, the sample point is displaced between the image to be encoded and the reference image. Therefore, in the latter case, the compression control unit 20 performs a setting for correcting the motion vector by motion compensation prediction.

  In addition, when sampling of sampling points is different between the luminance plane and the color difference plane of the image, the motion vector used for the motion compensation prediction with the luminance component cannot be directly applied to the motion compensation prediction of the color difference component. Therefore, also in this case, the compression control unit 20 performs a setting for correcting the motion vector in the motion compensation prediction.

  Step # 403: The encoding processing unit 22 encodes information on the header area of the image to be encoded. Here, information on the encoding format (I picture, P picture, B picture) of the image to be encoded, auxiliary information, and correction information are added to the header area of the image under the control of the compression control unit 20. Is done. For example, the incidental information includes an identifier indicating whether the image to be encoded is a first image or a second image. The accompanying information includes an identifier indicating the data format of the compression source image, an identifier indicating the sampling point thinning pattern in the luminance thinned image and the color difference thinned image, and the sampling point in the first image and the second image. An identifier indicating the extraction pattern may be included. The correction information is, for example, an identifier that indicates whether the above-described motion vector correction is necessary.

  In addition, for example, when the encoding processing unit 22 in # 403 performs compression encoding in accordance with the main profile of MPEG-2 video, the above-described incidental information and correction information are added to the user data area following the picture coding extension. You can include it.

  Step # 404: When generating the P picture or the B picture, the prediction information generation unit 40 of the encoding processing unit 22 performs motion compensation prediction on the macroblock using the reference image. Further, when generating an I picture, the prediction information generation unit 40 outputs macro block information as a through output.

  Step # 405: When generating the P picture or the B picture, the prediction unit 41 of the encoding processing unit 22 outputs the prediction value of the macroblock obtained by the motion compensation prediction (# 404). In addition, when generating an I picture, the prediction unit 41 outputs a zero value as a prediction value. Then, the subtraction unit 32 subtracts the prediction value from the original image signal and outputs a prediction error.

  Step # 406: The orthogonal transform unit 33 of the encoding processing unit 22 performs orthogonal transform on the prediction error (# 405).

  Step # 407: The quantization unit 34 of the encoding processing unit 22 quantizes the orthogonal transform coefficient (# 406) of the prediction error.

  Step # 408: The variable length coding unit 35 of the coding processing unit 22 performs variable length coding on prediction information indicating a motion vector, a reference image, and the like, and a quantization coefficient. In step # 408, encoded information for one block is output to the subsequent circuit.

  Step # 409: The inverse quantization unit 36 of the encoding processing unit 22 performs inverse quantization on the quantization coefficient.

  Step # 410: The inverse orthogonal transform unit 37 of the encoding processing unit 22 performs inverse orthogonal transform on the orthogonal transform coefficient obtained by inverse quantization and decodes the prediction error.

  Step # 411: The adding unit 38 of the encoding processing unit 22 adds the predicted value of the macroblock to the decoded prediction error to generate a local decoded value of the picture. The above-described local decoded value is stored in the second storage unit 39 of the encoding processing unit 22.

  Step # 412: The compression control unit 20 determines whether or not one frame has been encoded. If the above requirement is satisfied (YES side), the process returns to # 311. On the other hand, when the above requirement is not satisfied (NO side), the process returns to # 404 and the encoding processing unit 22 repeats the above operation. By the loop described above, the image to be encoded is encoded in units of blocks. Above, description of FIG. 23 is complete | finished.

  FIG. 24 is a flowchart showing an example of motion compensation prediction processing in # 404 of FIG.

  Step # 501: The prediction information generation unit 40 of the encoding processing unit 22 generates prediction information such as a macroblock motion vector, a prediction mode, and a reference image.

  When motion compensation prediction is performed between the first image and the second image, the prediction information generation unit 40 in # 501 corrects the displacement of the pixel position from the reference image so as to correct the motion vector of the macroblock. May be offset corrected.

  Step # 502: The prediction information generating unit 40 of the encoding processing unit 22 determines whether or not the macroblock is to be intraframe encoded (that is, whether or not to encode as a so-called intra macroblock). For example, in a case where the prediction error cannot be made smaller than a predetermined condition even if motion compensation prediction is performed, the prediction information generation unit 40 determines to encode the macroblock in the frame. If the above requirement is satisfied (YES side), the process returns to # 405 in FIG. In this case, the prediction information generation unit 40 outputs the macroblock information as a through output. On the other hand, if the above requirement is not satisfied (NO side), the process proceeds to # 503.

  Step # 503: The prediction information generation unit 40 of the encoding processing unit 22 performs motion compensation prediction on the luminance component of the macroblock based on the above-described prediction information (# 501).

  Step # 504: The prediction information generation unit 40 of the encoding processing unit 22 corrects the motion vector of the luminance component of the macroblock according to the color difference component of the macroblock. The prediction information generation unit 40 in # 504 corrects the motion vector according to the sampling difference of the sample points on the luminance plane and the color difference plane of the image.

  For example, in a YUV 4: 4: 4 format image, the centroids of the luminance component sample point and the color difference component sample point coincide. Therefore, in the case of the YUV 4: 4: 4 format, the prediction information generation unit 40 applies the motion vector of the luminance component of the macro block as it is to the prediction of the color difference component. In the YUV 4: 2: 2 format image, as described above, the motion vectors of the U and V channels are shifted by 0.5 pixels (MV [0.5, 0]) in the horizontal direction in terms of the Y channel. . Therefore, in the YUV 4: 2: 2 format, the prediction information generation unit 40 shifts the motion vector of the luminance component of the macroblock by 0.5 pixels in the horizontal direction and applies it to the prediction of the color difference component. In the YUV 4: 2: 0 format image, as described above, the motion vectors of the U and V channels are shifted by 0.5 pixels (MV [0.5, 0. 5]). Therefore, in the YUV 4: 2: 0 format, the prediction information generation unit 40 shifts the motion vector of the luminance component of the macroblock by 0.5 pixels in the horizontal direction and the vertical direction, and applies the prediction to the color difference component.

  Step # 505: The prediction information generation unit 40 of the encoding processing unit 22 performs motion compensation prediction on the color difference component of the macroblock based on the corrected prediction information (# 504) described above. Thereafter, the processing returns to # 405 in FIG. Above, description of FIG. 24 is complete | finished.

(Example of video decoding operation)
Hereinafter, an example of a moving image decoding operation according to the second embodiment will be described. FIG. 25 is a flowchart illustrating an example of a moving image decoding operation in the decoding processing unit 113 according to the second embodiment.

  Step # 601: The decoding processing unit 113 acquires compressed moving image information from the storage medium 110.

  Step # 602: The decoding unit 62 acquires supplementary information and correction information from the compressed video information. The incidental information and correction information are output to the decoding control unit 61.

  Step # 603: The decoding control unit 61 refers to incidental information and correction information. Thereby, the decoding control unit 61 includes information indicating whether or not the motion vector needs to be corrected, information on the thinning pattern of the sample points in the luminance thinned image and the color difference thinned image, and the composite pattern of the first image and the second image. Get information.

  Step # 604: The decoding unit 62 converts the first image (FIG. 3 (d), FIG. 4 (d), FIG. 5 (d), FIG. 6 (d)) is decoded. Also, the decoding unit 62 converts the second image (FIG. 3 (e), FIG. 4 (e), FIG. 5 (e), FIG. 6 () from the second encoded information in accordance with the main profile of MPEG-2 video, for example. e)) is decrypted. When motion compensation prediction is performed between the first image and the second image at the time of compression, the decoding unit 62 performs the correction of the motion vector based on the above-described correction information under the control of the decoding control unit 61. To do.

  Step # 605: The first separation unit 54 separates the input first image (# 604) into the Y channel and the U and V channels. The Y channel of the first image is output to the first combining unit 56, and the U and V channels of the first image are output to the second combining unit 57.

  Step # 606: The second separation unit 55 separates the input second image (# 605) into the Y channel and the U and V channels. The Y channel of the second image is output to the first combining unit 56, and the U and V channels of the second image are output to the second combining unit 57. Note that the process of # 605 and the process of # 606 may be executed in parallel.

  Step # 607: The first combining unit 56 combines the Y channel of the first image and the Y channel of the second image using the accompanying information. Note that the processing of # 607 corresponds to the processing of # 208 in FIG.

  Step # 608: The second combining unit 57 combines the U and V channels of the first image and the U and V channels of the second image using the accompanying information. Note that the processing of # 608 corresponds to the processing of # 209 in FIG.

  Step # 609: The first interpolation unit 58 inter-frame interpolates the sample points of the luminance thinned image restored in # 607. Note that the processing of # 609 corresponds to the processing of # 210 in FIG.

  Step # 610: The second interpolation unit 59 inter-frame interpolates the sample points of the color difference thinned image restored in # 608. Note that the processing of # 410 corresponds to the processing of # 211 in FIG.

  Step # 611: The image output unit 60 associates the luminance interpolated image and the color difference interpolated image of the same frame with one decoded frame (FIG. 9 (c), FIG. 10 (c), FIG. 11 (c), FIG. 12 (c)).

  Step # 612: The decoding control unit 61 determines whether or not the decoding of the entire compressed moving image information has been completed. If the above requirement is satisfied (YES side), the decoding control unit 61 ends the decoding operation. On the other hand, when the above requirement is not satisfied (NO side), the process returns to # 601 to decode the next frame. Above, description of FIG. 25 is complete | finished.

  FIG. 26 is a flowchart showing an example of the decoding process in # 604 of FIG.

  Step # 701: Based on the information acquired in # 603, the decoding control unit 61 determines the encoding format of the image to be decoded, the necessity of correcting the motion vector between the first image and the second image, the luminance component, Sets whether or not to correct a motion vector between color difference components.

  Step # 702: The code decoding unit 71 of the decoding unit 62 decodes a variable length code of a certain macroblock, and obtains prediction information indicating a motion vector, a reference image, and the like, and a quantization coefficient.

  Step # 703: The inverse quantization unit 72 of the decoding unit 62 inversely quantizes the quantization coefficient (# 702) of the macroblock.

  Step # 704: The inverse orthogonal transform unit 73 of the decoding unit 62 performs inverse orthogonal transform on the orthogonal transform coefficient obtained by inverse quantization (# 703) to decode the prediction error.

  Step # 705: When decoding the P picture or the B picture, the prediction unit 77 of the decoding unit 62 performs motion compensation prediction on the macroblock using the prediction information, and outputs a prediction value of the macroblock. Moreover, the prediction unit 77 outputs a zero value as a prediction value when decoding an I picture.

  Step # 706: The adding unit 74 of the decoding unit 62 adds the predicted value of the macroblock to the decoded prediction error to generate a local decoded value of the picture. As a result, one macroblock image is decoded. Note that the above-described local decoded value is stored in the second storage unit 76.

  Step # 707: The decoding control unit 61 determines whether or not one frame has been decoded. If the above requirement is satisfied (YES side), the process returns to # 605. On the other hand, when the above requirement is not satisfied (NO side), the process returns to # 702 and the decoding unit 62 repeats the above operation. According to the above loop, the encoded image is decoded in units of blocks. This is the end of the description of FIG.

  FIG. 27 is a flowchart showing an example of motion compensation prediction processing in # 705 of FIG.

  Step # 801: The prediction unit 77 of the decoding unit 62 sets a macroblock motion vector, a prediction mode, a reference image, and the like based on the decoded prediction information (# 702).

  Step # 802: The prediction unit 77 of the decoding unit 62 determines whether or not the macroblock to be decoded is an intra macroblock that is intraframe-encoded. If the above requirement is satisfied (YES side), the process returns to # 706 in FIG. In the case of # 802 on the YES side, the prediction unit 77 outputs a zero value as the predicted value as described above. On the other hand, if the above requirement is not satisfied (NO side), the process proceeds to # 803.

  Step # 803: The prediction unit 77 of the decoding unit 62 performs motion compensation prediction on the luminance component of the macroblock based on the above prediction information. When offset correction is performed during compression so as to cancel out the motion vector MV between the first image and the second image (# 501), the prediction unit 77 in # 803 corrects to remove the above-described offset. I do. For example, the prediction unit 77 may add the above-described motion vector MV to the motion vector of the luminance component of the macroblock. Thereby, in motion compensation prediction between the first image and the second image, the image can be correctly restored at the time of decoding.

  Step # 804: The prediction unit 77 of the decoding unit 62 corrects the motion vector of the luminance component of the macroblock according to the color difference component of the macroblock. The prediction unit 77 in # 804 corrects the motion vector according to the sampling difference of the sample points on the luminance plane and the color difference plane of the image. Note that the processing of # 804 is the reverse processing of # 504 of FIG. If the motion vector is not corrected at # 504 described above, the process at # 804 is omitted.

  Step # 805: The prediction unit 77 of the decoding unit 62 performs motion compensation prediction for the color difference component of the macroblock using the corrected prediction information (# 804) described above. Thereafter, the processing returns to # 706 in FIG. This is the end of the description of FIG.

(Operational effect of the second embodiment)
Hereinafter, operational effects in the second embodiment will be described. In the second embodiment, in addition to the effects of the first embodiment described above, the motion compensation prediction is performed between the first image and the second image, thereby improving the coding efficiency of the entire moving image. Can do. In the second embodiment, when motion compensation prediction is performed between the first image and the second image, the coding efficiency can be improved by correcting the motion vector. At this time, the moving image compression apparatus provides correction information indicating whether or not the above-described motion vector is corrected, so that the motion vector can be appropriately corrected at the video decoding apparatus side.

<Description of Third Embodiment>
FIG. 28 is a diagram illustrating an example of a device configuration according to the third embodiment. The apparatus of the third embodiment is a computer that functions as the above-described moving image compression apparatus and moving image decoding apparatus.

  A computer 201 illustrated in FIG. 28 includes a data reading unit 202, a storage device 203, a CPU 204, a memory 205, an input / output I / F 206, and a bus 207. The data reading unit 202, the storage device 203, the CPU 204, the memory 205 and the input / output I / F 206 are connected to each other via a bus 207. Furthermore, an input device 208 (a keyboard, a pointing device, etc.) and a monitor 209 are connected to the computer 201 via an input / output I / F 206. The input / output I / F 206 receives various inputs from the input device 208 and outputs display data to the monitor 209.

  The data reading unit 202 is used when reading uncompressed moving image information, the above-described compressed moving image information, and a program from the outside. For example, the data reading unit 202 communicates with a reading device (such as an optical disk, a magnetic disk, or a magneto-optical disk reading device) that acquires data from a removable storage medium, or an external device in accordance with a known communication standard. A communication device to perform (USB interface, LAN module, wireless LAN module, etc.).

  The storage device 203 is a storage medium such as a hard disk or a non-volatile semiconductor memory. The storage device 203 stores the above-described program and various data necessary for executing the program. The storage device 203 can also store compressed moving image information read from the data reading unit 202.

  The CPU 204 is a processor that comprehensively controls each unit of the computer 201. The CPU 204 functions as the image processing unit 111, the compression processing unit 112, and the decoding processing unit 113 of the first embodiment or the second embodiment by executing the program.

  The memory 205 temporarily stores various calculation results in the program. The memory 205 is, for example, a volatile SDRAM.

  Here, in the third embodiment, uncompressed moving image information is acquired from the data reading unit 202 or the storage device 203, and the compression operation shown in FIG. Alternatively, the above-described compressed moving image information is acquired from the data reading unit 202 or the storage device 203, and the CPU 204 performs the decoding operation shown in FIG. In the third embodiment, substantially the same effect as that of the first embodiment or the second embodiment can be obtained.

<Supplementary items of the embodiment>
(Supplement 1): In the above-described embodiment, the case of compressing an image in the YUV 4: 4: 4 format, the YUV 4: 2: 2 format, and the YUV 4: 2: 0 format has been described. However, the application of the present invention is not limited to these data formats.

  For example, the present invention may be applied to a YUV4: 0: 0 format image. This is the case of a monochrome image without U and V channel information. Therefore, only the Y channel of the compression source image needs to be encoded and decoded in the same manner as in the first embodiment or the second embodiment.

  The compression source image of the present invention may be an RGB image. For example, in the case of an RGB 4: 4: 4 format image, encoding and decoding may be performed in the same manner as in the first or second embodiment, assuming that G is a luminance plane and R and B are color difference planes.

  (Supplement 2): In the above-described embodiment, an example of compression encoding based on the main profile of MPEG-2 video has been described. However, in the present invention, JPEG, AVC, or other compression encoding may be applied, or non-compression encoding may be applied.

  (Supplement 3): The attachment of the incidental information indicating the pattern of image thinning is not limited to the example of the above-described embodiment. For example, by predefining a frame operation rule indicating a thinning pattern that can be determined from temporal reference information of MPEG-2 video, an image thinning pattern may be implicitly notified to the decoding processing apparatus. it can.

  (Supplement 4): In the above-described embodiment, the case where the compression source image is a moving image has been described. However, the present invention is naturally applicable to still image compression.

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

DESCRIPTION OF SYMBOLS 11 ... Color component separation part, 12 ... 1st thinning part, 13 ... 2nd thinning part, 14 ... 1st separation part, 15 ... 2nd separation part, 16 ... 1st image generation part, 17 ... 2nd image generation part , 18 ... 1st encoding part, 19 ... 2nd encoding part, 20 ... Compression control part, 21 ... Multiplexing part, 22 ... Encoding process part, 51 ... Demultiplexing part, 52 ... 1st decoding part, 53 ... 2nd decoding part, 54 ... 1st separation part, 55 ... 2nd separation part, 56 ... 1st synthetic | combination part, 57 ... 2nd synthetic | combination part, 58 ... 1st interpolation part, 59 ... 2nd interpolation part, 60 DESCRIPTION OF SYMBOLS Image output part 61 ... Decoding control part 62 ... Decoding part 100 ... Electronic camera 101 ... Imaging optical system 102 ... Imaging element 103 ... Signal processing part 104 ... Image processing engine 105 ... 1st memory, 106 ... Second memory, 107 ... Recording I / F, 108 ... Monitor, 109 ... Operation unit, 110 ... Storage medium, 11 DESCRIPTION OF SYMBOLS ... Image processing part, 112 ... Compression processing part, 113 ... Decoding processing part, 201 ... Computer, 202 ... Data reading part, 203 ... Storage device, 204 ... CPU, 205 ... Memory, 206 ... Input / output I / F, 207 ... Bus 208 ... Input device 209 ... Monitor

Claims (8)

Generate a thinned image by discretely extracting sample points along the row direction from the original image so that the positions of the sample points on the odd-numbered rows and the sample points on the even-numbered rows are shifted in the row direction. A thinning part;
A first image generation unit that extracts the sample points of the odd-numbered rows from the thinned image and generates a first image;
A second image generation unit that extracts the sample points of the even-numbered rows from the thinned image and generates a second image;
An encoding unit for encoding each of the first image and the second image;
An output unit that outputs the encoded information of the first image and the encoded information of the second image in association with auxiliary information indicating a decoding method for the thinned image;
An image compression apparatus comprising: The image compression apparatus according to claim 1.
The compression source image is a color image,
The thinning unit is an image compression device that extracts the sample points with different patterns for each component of color space from the compression source image to generate the thinned image. The image compression apparatus according to claim 1 or 2,
The compression source image is a moving image,
The thinning unit is an image compression device that generates the thinned image in each frame of the moving image. The image compression apparatus according to claim 3.
The encoding unit is an image compression apparatus that performs motion compensation prediction between a plurality of first images generated from different frames or motion compensation prediction between a plurality of second images generated from different frames. In the image compression device according to any one of claims 1 to 4,
The encoding unit is an image compression apparatus that performs motion compensation prediction between the first image and the second image. The image compression apparatus according to claim 5.
The encoding unit is an image compression device that corrects a motion vector at the time of motion compensation prediction according to a difference in sample points between the first image and the second image. The image compression apparatus according to any one of claims 1 to 6,
The compression source image is an image compression device in any one of YUV4: 4: 4 format, YUV4: 2: 2 format, YUV4: 2: 0 format, YUV4: 0: 0 format, or RGB4: 4: 4 format. Generate a thinned image by discretely extracting sample points along the row direction from the original image so that the positions of the sample points on the odd-numbered rows and the sample points on the even-numbered rows are shifted in the row direction. Thinning process,
A first image generation process for generating a first image by extracting the sample points of the odd-numbered rows from the thinned image;
A second image generation process for extracting a sample point of the even-numbered row from the thinned image and generating a second image;
An encoding process for encoding each of the first image and the second image;
An output process in which the encoded information of the first image and the encoded information of the second image are associated with and output additional information indicating a decoding method for the thinned image;
A program that causes a computer to execute.

Images