JP4960400B2 - Stereo image encoding method and stereo image decoding method - Google Patents

Description

  The present invention relates to an encoding method for efficiently predictive encoding a stereo image and a decoding method for decoding the encoded code data.

  A stereo (stereoscopic) image (hereinafter, referred to as a stereo image) is an image intended to provide a stereoscopic effect visually by displaying a left-eye image and a right-eye image so that they can be seen by the left and right eyes, respectively. Therefore, when display is performed at the same resolution and frame rate, twice as much image data is required as compared to a normal planar image.

  Existing transmission channels (for example, broadcast, storage media, wired or wireless communication lines) are supposed to transmit flat images and send twice as much image data in parallel and in synchronization. I can't. For this reason, the image data increased by a factor of two is reduced to ½ and sent over an existing transmission channel.

  For example, H.M. The Stereo video information SEI (Supplemental Enhancement Information) message employed in H.264 / MPEG-4 AVC records additional information indicating left and right images alternately transmitted in frame units or field units in a bit stream.

  In addition, US Patent Application Publication No. 2003/0223499 (in Patent Document 1, after subsampling the left eye image and the right eye image at a grid-like position that interpolates each other, squeezing in the horizontal direction, the left eye image is displayed on the left side of the screen. A stereo image format is disclosed in which the right and left eye images are arranged in the right half of the screen, and this stereo image format is a concatenation of a single left and right image having the same number of pixels as the left or right eye image before sub-sampling. This is an image (hereinafter, this image format is referred to as a “side-by-side method”).

  Further, for example, there is a stereo image format in which the left eye image and the right eye image are subsampled at a grid-like position for interpolating each other, and then the left eye image and the right eye image are merged in units of pixels (hereinafter, this image format is referred to as “pixel interleaving”). Called "method"). Furthermore, after subsampling and squeezing each of the left-eye image and the right-eye image in the vertical direction, the left and right images, such as a stereo image format arranged vertically, are subsampled to 1/2 the number of pixels in the space plane, There are various stereo image formats in which both are combined so as to have the same number of pixels as one image data.

US Patent Application Publication No. 2003/0223499

  However, in the stereo image format described in the above patent document, there is a parallax between the left-eye image and the right-eye image, so when performing predictive coding while being connected to one image without distinguishing them, The compression efficiency becomes worse. Further, when a color signal with parallax is encoded by an encoding method including orthogonal transformation, color blur occurs.

  The present invention has been invented to solve these problems in view of the above points, and efficiently predictively encodes a stereo image of a type in which left and right parallax images are connected to one image. The purpose is to do.

  In order to solve the above-described problems and achieve the object, the present invention provides a first parallax image having a pixel at the first phase position of the checkerboard pattern and a pixel at the second phase position of the checkerboard pattern. From encoded data obtained by predictively encoding a picture in which the first parallax image and the second parallax image are combined in a stereo image signal having a second parallax image having (A) the parallax Prediction residual for each image, (B) prediction mode information for each parallax image, and (C) combined information indicating an arrangement of the first parallax image and the second parallax image on the picture. With reference to a decoding step for decoding, each pixel value on the already decoded parallax image, and a pixel value interpolated from neighboring pixels at positions where no pixel exists on the already decoded parallax image, Prediction mode A prediction signal generation step for generating a prediction signal for each parallax image according to the information, and a decoded image generation step for generating a decoded image signal by adding the prediction signal and the prediction residual according to the combination information. It is characterized by.

  In order to solve the above-described problems and achieve the object, the present invention also provides a first parallax image having a pixel at a first phase position of the checkerboard pattern and a second phase position of the checkerboard pattern. A prediction signal of each parallax image of a picture obtained by combining the first parallax image and the second parallax image of a stereo image signal having a second parallax image having pixels in a plurality of prediction modes. According to the selected prediction mode, each pixel value on an already encoded parallax image and a pixel value interpolated from neighboring pixels at positions where no pixel exists on the already encoded parallax image are generated. A prediction signal generation step; a residual generation step for generating a prediction residual between each parallax image and the prediction signal; (A) the prediction residual for each parallax image; and (B) the selection for each parallax image. The And (C) an encoding step for encoding combined information indicating an arrangement of the first parallax image and the second parallax image on the picture, and decoding the prediction residual A decoded image for generating a decoded image signal by adding the decoded prediction residual to the prediction signal based on the decoding step for generating a decoded prediction residual and information related to the prediction mode and the combined information And a generating step.

  According to the stereo image encoding method and the stereo image decoding method of the present invention, it is possible to efficiently predictively encode a stereo image of a type in which left and right parallax images are connected to one image. .

FIG. 1 is a diagram illustrating an example of a functional configuration of an image encoding device 20 for moving image encoding according to the present embodiment. FIG. 2 is a diagram illustrating an example of a detailed functional configuration of the predicted image creator 115. FIG. 3 is a diagram illustrating a pixel interleaving method. FIG. 4 is a diagram illustrating a squeezed stereo image. FIG. 5 is a diagram illustrating an example of a prediction process for an interleaved stereo image. FIG. 6 is a diagram illustrating an example of prediction processing in the intra-frame predictor 1151 for a side-by-side stereo image. FIG. 7 is a diagram illustrating an example (part 1) of the prediction process in the inter-frame predictor 1152 for the pixel interleaved stereo image. FIG. 8 is a diagram illustrating an example (part 2) of the prediction process in the inter-frame predictor 1152 for the side-by-side stereo image. FIG. 9 is a diagram illustrating an example of deblocking filter processing in the loop filter 113 for a pixel interleaved stereo image. FIG. 10 is a diagram illustrating another example of deblocking filter processing in the loop filter 113 for a side-by-side stereo image squeezed in the horizontal direction. FIG. 11 is a diagram illustrating a data structure of information relating to a stereo image format included in encoded data of a stereo image. FIG. 12 is a diagram illustrating an overall data structure of encoded data of a stereo image. FIG. 13 is a diagram illustrating a data structure of information for each predetermined unit (slice) of encoding in encoded data of a stereo image. FIG. 14 is a diagram illustrating a data structure of information for each predetermined unit (macroblock) of encoding in encoded data of a stereo image. FIG. 15 is a diagram illustrating a data structure including prediction mode information for each macroblock in encoded data of a stereo image. FIG. 16 is a diagram illustrating an example of sub macroblock prediction modes and syntax for motion vectors in encoded data of a stereo image. FIG. 17 is a flowchart for explaining an image coding method according to the present embodiment. FIG. 18 is a flowchart showing a part of the processing performed in step S103. FIG. 19 is a diagram showing an image decoding apparatus according to the present embodiment. FIG. 20 is a diagram illustrating an example of a detailed functional configuration of the predicted image creator 204. FIG. 21 is a flowchart for explaining an image decoding method according to the present embodiment. FIG. 22 is a flowchart showing a part of the processing performed in step S201.

  Hereinafter, the present embodiment will be described with reference to the drawings. In the following embodiments, “sampling” of pixels and “subsampling” of pixels refer to extracting some pixels from a plurality of pixels.

[Embodiment]
FIG. 1 is a diagram illustrating an example of a functional configuration of an image encoding device 20 for moving image encoding according to the present embodiment. The image encoding device 20 compresses the input image signal 100 for each frame to generate encoded data 117.

  The image encoding device 20 includes an input frame buffer 118, a differential signal generator 101, an orthogonal transformer 104, a quantizer 106, an inverse quantizer 109, an inverse orthogonal transformer 110, a locally decoded image signal generator 111, a loop filter. 113, a frame memory 114, a predicted image generator 115, and an entropy encoder 108.

  The input image signal 100 is input in units of frames, for example. The input frame buffer 118 temporarily stores the input image signal 100 and outputs it to the differential signal generator 101. The difference signal generator 101 calculates a difference between the input image signal 100 and the predicted image signal 102 and generates a prediction error signal 103. An input image signal 100 that is different from the predicted image signal 102 is a parallax separation signal separated into two parallaxes.

  The orthogonal transformer 104 performs orthogonal transformation on the prediction error signal 103. The orthogonal transform is, for example, a discrete cosine transform (hereinafter referred to as “DCT”). Orthogonal transformation coefficient information 105 is obtained by the orthogonal transformation. The orthogonal transform coefficient information 105 is, for example, a DCT coefficient.

  The orthogonal transformer 104 performs orthogonal transformation on each of the left and right parallax images for each rectangle composed of pixels of the same parallax image. This rectangle is, for example, a block or a macroblock.

  For example, with respect to one parallax image, the orthogonal transformer 104 generates a pixel at a pixel position where no pixel exists among the pixel positions included in the frame, and performs orthogonal transformation for each rectangle including the predicted pixel. Do. The code data generated by the image encoding device 20 includes codes related to a predetermined number of pieces of orthogonal transform coefficient information from the low frequency side among the orthogonal transform coefficient information 105 generated by the orthogonal transform. Good. Thereby, the redundant part increased by the pixel produced | generated by prediction can be reduced. For example, half the number of orthogonal transform coefficient information may be deleted, and the same number of orthogonal transform coefficient information 105 as the number of pixels before interpolation may be output.

  For example, the orthogonal transformer 104 squeezes a pixel position where no pixel exists for one parallax image, and then, for each pixel group corresponding to a pixel position having a phase in the same picture in a direction different from the squeezed direction. Then, orthogonal transformation is performed.

  More specifically, when squeezing in the horizontal direction, each of a predetermined unit composed of pixels belonging to the odd lines and a predetermined unit composed of pixels belonging to the even lines is a unit for performing orthogonal transformation. In addition, when squeezing in the vertical direction, a predetermined unit composed of pixels belonging to odd columns and a predetermined unit composed of pixels belonging to even columns are set as units for performing orthogonal transformation. The predetermined unit includes a plurality of pixels arranged in a rectangle, and includes, for example, the same number of pixels as a block or a macroblock.

  In this orthogonal transform, since no pixel interpolation is performed, there is no increase in redundancy due to an increase in the number of pixels. Therefore, the generated code data may include codes related to all orthogonal transform coefficient information 105 generated by this orthogonal transform.

  The quantizer 106 quantizes the orthogonal transform coefficient information 105 and outputs quantized orthogonal transform coefficient information 107. The quantized orthogonal transform coefficient information 107 is bifurcated, and one is guided to the entropy encoder 108. On the other hand, the inverse quantizer 109 and the inverse orthogonal transformer 110 sequentially receive the process opposite to the process of the quantizer 106 and the orthogonal transformer 104 to obtain a signal similar to the prediction error signal, and then the local signal. The decoded image signal generator 111 adds the predicted image signal 102 to generate a locally decoded image signal 112.

  The loop filter 113 performs a filter process on the locally decoded image signal 112. The filtered local decoded image signal 112 is stored in the frame memory 114. The filter processing in the loop filter 113 will be described later. Note that the processing by the loop filter 113 may be omitted.

  The predicted image creator 115 generates a predicted image signal based on the prediction mode information from the input image signal 100 and the locally decoded image signal 112. The prediction mode information includes information indicating which prediction mode of a plurality of prediction modes is used. The prediction mode is, for example, H.264. It is a plurality of prediction modes of intra prediction in H.264 / MPEG-4 AVC. The prediction mode is also, for example, H.264. It is a plurality of prediction modes between pictures in H.264 / MPEG-4 AVC. Details of the predicted image generator 115 and a method of generating a predicted image signal will be described later.

  The generated predicted image signal 102 is output from the predicted image generator 115 together with information 116 including motion vector information, prediction mode information, and stereo image format information of the predicted image signal.

  Stereo image format information is information indicating the arrangement of pixels for each parallax image when image data including a plurality of parallax images is stored in a data format of image data including a single parallax image. Stereo image formats include an interleave method and a side-by-side method.

  In the entropy encoder 108, information 116 including quantized orthogonal transform coefficient information 107, motion vector information, prediction mode information, and stereo image format information is entropy-encoded, and encoded data 117 generated thereby is shown in FIG. Not sent to transmission system or storage system.

  FIG. 2 is a diagram illustrating an example of a detailed functional configuration of the predicted image creator 115. The predicted image creator 115 includes a mode determiner 1150, an intra-frame predictor 1151, an inter-frame predictor 1152, a parallax image sampling position interpolator 1154, and a parallax image separator 1155.

  The parallax image separator 1155 separates the local decoded image signal 112 and the input image signal 100 into signals of only pixels belonging to each parallax image according to the sampling method and merge method of the stereo image to be processed. If it is done, cancel it.

  The parallax image sampling position interpolator 1154 interpolates pixels at positions that were not sampled for each parallax in the local decoded image signal 112 separated for each parallax image and the input image signal 100. The interpolated input image signal 100 is output to a motion vector detector 1153, and the interpolated local decoded image signal 112 is output to an intra-frame predictor 1151, an inter-frame predictor 1152, and a motion vector detector 1153.

  The intra-frame predictor 1151 creates a predicted image signal based on the intra-frame prediction from the locally decoded image signal 112 of the area already encoded in the frame being processed, which is stored in the frame memory 114.

  The inter-frame predictor 1152 performs motion compensation on the reproduced image signal stored in the frame memory 114 based on the motion vector detected by the motion vector detector 1153, and creates a predicted image signal based on the inter-frame prediction. To do.

  The intra-frame predictor 1151 performs processing in M intra-frame prediction modes, and the inter-frame predictor 1152 performs processing in N inter-frame prediction modes.

  More specifically, the prediction processing performed by the intra-frame predictor 1151 and the inter-frame predictor 1152 is performed separately for each parallax image in which the separation or squeezing of the coding unit is released, and the predicted image signal is also parallax. Every time, the signal from which separation or squeezing is released is output.

  The mode determiner 1150 receives the output of the intra-frame predictor 1151 and the output of the inter-frame predictor 1152, and selects a prediction mode. The mode determiner 1150 receives a prediction image signal based on one prediction mode selected from N inter-frame prediction modes or a prediction image signal based on one prediction mode selected from M intra-frame prediction modes. Output.

  The mode determiner 1150 determines a prediction mode for each parallax image, and finally outputs a prediction image signal 102 in which the prediction image signal from which separation or squeezing has been released is merged or squeezed into the original stereo image format. .

  3 and 4 are diagrams showing examples of stereo image types subsampled in a grid pattern. FIG. 3 is a diagram illustrating a pixel interleaving method. In FIG. 3, the left-eye image L1 and the right-eye image R1 are subsampled in a lattice shape at positions where the phases are different from each other. The left eye image L1 and the right eye image R1 are interleaved pixel by pixel to form one stereo image S1. The stereo image S1 always has the same vertical or horizontal phase between adjacent pixels, but has a different parallax. Note that a pattern including two sets of pixel positions sub-sampled in a lattice shape at positions having different phases from each other is referred to as a checkerboard pattern. The stereo image S1 has a pixel of the left eye image at the position of the first phase of the checkerboard pattern and a pixel of the right eye image at the position of the second phase of the checkerboard pattern.

  FIG. 4 is a diagram illustrating a squeezed stereo image. The left-eye image L2 and the right-eye image R2 in FIG. 4 are subsampled in a grid pattern at positions having different phases as in FIG. In the stereo image S21 of FIG. 4, pixels belonging to the left-eye image L2 are arranged in the left half of the frame, and pixels belonging to the right-eye image R2 are arranged in the right half of the frame. The stereo image S21 is a side-by-side stereo image. The stereo image S21 has the same phase in the vertical direction between pixels arranged in the horizontal direction, but has a different phase in the horizontal direction.

  In the stereo image S22, pixels belonging to the right eye image R2 are arranged in the upper half of the frame, and pixels belonging to the left eye image L2 are arranged in the lower half of the frame. The stereo image S22 has the same horizontal phase between pixels arranged in the vertical direction, but has a different vertical phase.

  FIG. 5 is a diagram illustrating an example of a prediction process in the intra-frame predictor 1151 for an interleaved stereo image. In FIG. 5, a rectangle indicates the block B for which prediction is performed, and a circle indicates a pixel included in the parallax image. In the block B, only the pixel for the left eye is shown, and an example in which the pixel for the left eye is predicted vertically from the adjacent pixels outside the block B is shown.

  The adjacent pixels are interleaved with pixels for the right eye for each pixel. For the prediction, an image obtained by interpolating the pixel at the pixel position for the right eye using the adjacent pixel for the left eye is used as a reference image. For the interpolation pixel, for example, an average value of two adjacent left-eye pixels may be used as the pixel value of the interpolation pixel. In FIG. 5, a circle with hatching is an interpolation pixel.

  Further, for example, for one pixel position, interpolation may be performed by performing filter processing using six pixels of the left-eye image adjacent to the left or right or up and down. More specifically, for example, a total of 6 pixels are selected for each pixel position, 3 pixels in each of the left and right directions. From these six pixels, H. The interpolation may be performed by a 6-tap filter (1/32, −5/32, 20/32, 20/32, −5/32, 1/32) for 1/2 pixel interpolation used in H.264.

  If the reference pixel position is at the edge of the screen and the adjacent pixel necessary for interpolation cannot be obtained, for example, the pixel value at the end may be interpolated as the pixel value of the pixel outside the screen. As a symmetry point, the pixels in the screen may be mirror-copied outside the screen and used for interpolation. Prediction within the block is performed using the reference pixels obtained by interpolation in this way.

  In FIG. 5, when predicting in the vertical direction, for example, the value of the reference pixel adjacent to the upper end of the block is used as it is as the predicted value. Similarly for the right-eye pixels in the block, the left-eye pixel positions of every other reference pixel are interpolated using the adjacent right-eye pixels, and the right-eye pixel positions in the block are used as reference pixels. Make a prediction.

  At this time, the prediction method (prediction mode) of the pixel for the left eye and the pixel for the right eye may be different. Here, an example of performing prediction in the vertical direction has been described. Note that the prediction may be performed by any method as long as the prediction is performed by using a reference pixel string obtained by interpolating pixels at unsampled positions with pixels belonging to the same parallax image. For example, H.C. H.264 intra prediction or the like may be applied.

  FIG. 6 is a diagram illustrating an example of prediction processing in the intra-frame predictor 1151 for a side-by-side stereo image. 6 shows a pixel arrangement of one parallax image that is squeezed side-by-side in the horizontal direction, and f2 in FIG. 6 shows a pixel arrangement of one parallax image before being squeezed. .

  FIG. 6 shows an example in which prediction is performed in the vertical direction. In f2, pixels included in one parallax image are indicated by white circles and circles with lattice hatching. At f2, pixel positions P01 through P04 that have not been extracted by sampling are interpolated using adjacent sampled pixels P10 through P13. In FIG. 6, the interpolated pixels are hatched with dots. Then, using the reference pixel columns P01 to P04 and P10 to P14, prediction is performed by setting the reference pixel value as a predicted value in the vertical direction.

  Further, for example, when prediction is performed in the horizontal direction, since squeezing is not performed in the vertical direction, the reference pixel may be used for prediction as it is without interpolation. Also, for example, the positions of the reference pixels (pixels indicated by white circles in the figure) in the 0th row and the 2nd row that are 2 pixels away from the end of the block before squeezing are set to the white circle pixels and the 1st and 3rd rows, for example Alternatively, the pixel interpolated using the upper left pixel may be used as a reference pixel column.

  In addition, when performing intra-frame prediction on a stereo image squeezed in the vertical direction side-by-side, pixel positions that have not been sampled in the pixel arrangement before squeezing of the reference pixel adjacent to the left of the block are adjacent to each other. By using the reference pixel row interpolated using the sampled pixels, prediction in the horizontal direction can be performed.

  Note that the prediction is not limited to prediction in the horizontal direction and vertical direction, and any prediction method can be used as long as it is a method used for prediction by interpolating pixels at positions not sampled in the pixel arrangement before squeezing. Good. For example, H.C. H.264 intra prediction or the like may be applied.

  FIG. 7 is a diagram illustrating an example of a prediction process in the inter-frame predictor 1152 for a pixel interleaved stereo image. In FIG. 7, f31 and f32 are reference images, and f30 is an input image to be encoded. The reference image is separated into a left-eye image f31 and a right-eye image f32, and a pixel that has not been sampled is interpolated using surrounding pixels. In f31 and f32, pixels indicated by black circles and black squares are interpolated pixels.

  Using only the pixels belonging to the respective parallax images in the block b30, which is the encoding unit of the input image f30, a predicted image is obtained for the reference image separated and interpolated for each parallax. The predicted image is obtained by, for example, obtaining a motion vector for each parallax image using a technique such as block matching, and again sampling the reference image at the position indicated by the motion vector in a complementary grid pattern for each parallax. A predicted image is obtained by interleaving pixels with different parallaxes at positions that have not been performed.

  In addition, when calculating the motion vector with 1/2 pixel accuracy or 1/4 pixel accuracy, a position that is separated for each parallax and not sampled, such as f31 and f32 in FIG. In the supplemented reference image, a reference image obtained by interpolating the ½ pixel position or the ¼ pixel position is used. For example, H.H. For example, an H.264 interpolation filter may be used.

  FIG. 8 is a diagram illustrating an example different from FIG. 7 of the prediction processing in the inter-frame predictor 1152 for the side-by-side stereo image. In FIG. 8, f41 is a part of the reference image (pixels corresponding to the left-eye image are shown), and f40 is a part of the input image to be encoded. In f41, a circle with dot hatching is an interpolation pixel.

  In the reference image f41, squeezing is canceled, and pixels at positions where sampling is not performed are interpolated using peripheral pixels. The input image f40 shows pixels in a state where squeeze is released. For each encoding unit of the input image, for each reference image interpolated for each parallax, a motion vector is obtained for each parallax image using a technique such as block matching, and the position indicated by each motion vector The reference image is again sampled in a lattice pattern complementarily for each parallax and squeezed to obtain a predicted image.

  In addition, when the motion vector is calculated with 1/2 pixel accuracy or 1/4 pixel accuracy, a reference image such as f41 in which the squeezed is canceled and the unsampled position is complemented using the peripheral pixels. A reference image obtained by interpolating the ½ pixel position and the ¼ pixel position is used. This interpolation is, for example, H.264. For example, an H.264 interpolation filter may be used.

  Next, a detailed example of the loop filter 113 will be described. FIG. 9 is a diagram illustrating an example of deblocking filter processing in the loop filter 113 for a pixel interleaved stereo image. In FIG. 9, pixels of the first parallax image are indicated by circles, and pixels of the second parallax image are indicated by squares.

  Among the loop filters, the deblocking filter is a filter for reducing block distortion that occurs at the boundary between coding units. When the deblocking filter is applied in a state where the parallax images are interleaved with pixels, the effect of removing block distortion may not be obtained when a plurality of parallaxes exist in a pixel used for one filter process. Therefore, an effect of removing block distortion without an influence between parallaxes can be obtained by performing filter processing using every other pixel on the boundary edges in the vertical and horizontal directions. For example, a low-pass filter may be used as the deblocking filter.

  FIG. 10 is a diagram illustrating another example of deblocking filter processing in the loop filter 113 for a side-by-side stereo image squeezed in the horizontal direction. Since pixels with the same parallax are arranged in the horizontal direction, a filter is applied using the pixel columns as they are. However, in the vertical direction, pixels whose phases are shifted by one pixel are arranged in every row, so every other pixel. Filter processing is performed using the pixel columns.

  In addition, when squeezed in the vertical direction, the filtering process may be performed using a pixel column with one pixel skipped in the horizontal direction and the pixel column as it is in the vertical direction. In the case of the side-by-side method, the filtering process may not be performed on the line that is a boundary between parallaxes.

  As an example of the loop filter, the deblocking filter with respect to the boundary of the coding unit has been described. However, another filter process (for example, a Wiener filter process that approximates the input image) may be performed. In this case, for pixel interleaved stereo images, filter processing is performed on pixel columns or pixel blocks separated for each parallax, and for side-by-side stereo images, squeezing is canceled on pixel columns or pixel blocks. It is desirable to apply a filtering process to the filter. For any loop filter, it is preferable to perform a filtering process using a plurality of pixels having the same parallax and the same vertical or horizontal phase of the pixel position.

  Note that the loop filter 113 is not limited to a deblocking filter, and may be a filter that performs a filter process on a decoded image serving as a reference image. The filter processing by the loop filter 113 is not limited to the block boundary, and may be performed on the pixels included in the decoded stereo image.

  In this embodiment, an In-Loop filter that performs filter processing on a decoded image serving as a reference image will be described. However, a post filter that performs filter processing on an output decoded image separately from the reference image is also a loop. Similar to the filter 113, it is preferable to perform a filtering process on a pixel belonging to each parallax image by a plurality of pixels including pixels having the same vertical or horizontal phase of the pixel position.

  Next, an example of the orthogonal transformation method in the orthogonal transformer 104 is shown. When parallax images are interleaved with pixels, orthogonal transformation is not efficiently performed because a plurality of parallax pixels are included in one orthogonal transformation unit. Further, in the case of the side-by-side format, since the phase of one pixel is shifted for each row or column, the orthogonal transformation is not efficiently performed.

Therefore, two examples for efficiently performing orthogonal transform are shown.
In the first example, the residual signal is separated for each parallax image, or squeezing is canceled, and orthogonal transformation is performed on a signal obtained by interpolating a position not sampled from the peripheral pixel position. In this case, since the transform coefficient is doubled and the amount of information is increased, for example, only half of the low-frequency side transform coefficients are encoded by the entropy encoder 108 among the generated transform coefficients. Good. Note that the low-frequency side coefficient is a coefficient located in the upper left region of the quantization unit.

  In the second example, in the case of a pixel interleaved stereo image, the residual signal is separated for each parallax image, further divided into even-numbered lines and odd-numbered lines, and orthogonal transformation is performed on each block. In the case of the side-by-side format, when horizontally squeezed, it is divided into even-numbered and odd-numbered blocks, and when vertically squeezed, it is divided into even-numbered and odd-numbered blocks and orthogonal to each other. Apply conversion. In the case of the second example, since the number of transform coefficients does not increase, all of the output transform coefficients may be encoded by the entropy encoder 108.

  The motion vector and prediction mode information output from the predicted image generator 115, that is, the motion vector output from the motion vector detector 1153 and the prediction mode information indicating the prediction mode selected by the mode determiner 1150 are entropy encoded. Sent to the device 108, entropy-coded, and multiplexed into the encoded data 117.

  FIGS. 11 to 16 are diagrams illustrating examples of the format of a stereo image and a data structure in which motion vector information and prediction mode information are multiplexed.

  FIG. 11 shows an example of a data structure in which information indicating how stereo images are subsampled and merged in the spatial direction is multiplexed. This information is multiplexed in sequence units or picture units.

  In FIG. 11, spatial_subsampling_direction_info is information indicating the direction of sampling. For example, when this value is 01, it indicates that it is subsampled in the horizontal direction, when it is 10, when it is vertical, and when it is 11, it is subsampled in a grid pattern. left_subsampling_position_flag and right_subsampling_position_flag indicate in which phase the left and right parallax images are subsampled. For example, when sub-sampling is performed in the horizontal direction, if 0, the top field is used, and if it is 0, the bottom field is used. When sub-sampling is performed in the vertical direction, even columns are used if 0 and odd columns are used. In the case of sub-sampling in a grid pattern, 0 indicates sub-sampling in order from the upper left pixel, and 1 indicates sub-sampling from the first pixel on the upper left.

  squeezing_flag indicates whether or not the sampled parallax image is squeezed. For example, 0 indicates merging without squeezing, and 1 indicates squeezing and merging.

  When squeezing_flag is 0, that is, when squeezing is not performed, if left_subsumming_position_flag and right_subsampling_position_flag are the same value, it is not possible to fill in a position where sampling is not performed unless the phase of either parallax image is shifted. For this reason, the value of shift_view_flag indicating which parallax image is shifted and compensated is included in the encoded data. When this value is 0, for example, it indicates that the position of the right eye image is shifted, and when it is 1, for example, it indicates that the position of the left eye image is shifted and compensated.

  On the other hand, when squeezing_flag is 1, that is, when squeezed, squeezing_direction_flag indicating which direction is squeezed is further added. For example, when this value is 0, it indicates that it is squeezed in the horizontal direction, and when it is 1, it indicates that it is squeezed in the vertical direction.

  Further, left_image_topleft_flag is a flag indicating in which direction it is connected. For example, when this value is 0, the left-eye image is left (when squeezed in the horizontal direction) or upward (when squeezed in the vertical direction). 1 indicates that the right-eye images are connected so as to be left (when squeezed in the horizontal direction) or above (when squeezed in the vertical direction).

  When the above information is multiplexed into a bit stream and the sampling method is a lattice from these pieces of information, using the sampling phase, presence / absence of squeeze, etc., the above-described prediction image creator 115, loop filter 113, orthogonal The converter 104 performs a parallax image separation process, a squeeze release process, an interpolation process for unsampled pixel positions, and the like.

  Next, an example of multiplexing motion vector information and prediction mode information will be described. Here, as an example, H.P. 2 shows an example of a data structure when multiplexing motion vector information and prediction mode information when intra-frame prediction and inter-frame prediction similar to H.264 are performed.

  FIG. 12 is a diagram illustrating a syntax configuration of a bit stream that is encoded data output from the encoding apparatus. In this syntax structure example, the access unit (301) is a unit that is read in the decoding process, and the decoding process is performed for each unit. In the access unit (301), high-level syntax (302), slice layer syntax (305), and the like are packed according to the processing content and the coding structure. The high level syntax (302) is packed with syntax information of higher layers above the slice. For example, information regarding the stereo image format is stored in this layer. The high-level syntax has a sequence parameter set syntax (303) and a picture parameter set syntax (304).

  The slice layer syntax (305) specifies information necessary for each slice. The slice layer syntax (305) includes a slice header syntax (306) and a slice data syntax (307). The slice data syntax (307) includes a macroblock layer syntax (308) in which information necessary for decoding a macroblock layer included in the slice is specified, and a multi-block layer (309) for a stereo image subsampled in a grid pattern. ), And the change value of the quantization parameter and the mode information required for each macroblock are specified.

  The above-described syntax is an indispensable component at the time of decoding. If the syntax information is missing, there is a case where data cannot be correctly restored at the time of decoding.

  13 to 16 are diagrams illustrating examples of data structures for multiplexing motion vector information and prediction mode information.

  FIG. 13 is a diagram illustrating an example of the syntax of slice data. The mb_skip_flag, macroblock_layer (), and macroblock_layer_for_checkerboard () are included in the encoded data in the same number as the number of macroblocks in the slice.

  mb_skip_flag is a flag indicating whether or not decoding is possible from the previous encoding and decoding information without specifying any information necessary for decoding the macroblock. In the case of TRUE, the mb_skip_flag is below the macroblock layer syntax. The information may not be multiplexed. In the case of FALSE, first, the value of spatial_subsampling_direction_info indicating the sampling method of the target image is determined.

  When the value of spatial_subsampling_direction_info is a value indicating that subsampling is performed in a lattice shape, for example, 11, the macroblock layer syntax macrocube_layer_for_checkerboard () specifying information necessary for decoding the macroblock included in the slice is provided. The data are sequentially transmitted until end_of_slice_flag becomes 1.

  On the other hand, when the spatial_subsampling_direction_info is not a value indicating that the sub-sampling is performed in a lattice shape, for example, H.264. The same macroblock_layer () used in H.264 is sequentially transmitted until end_of_slice_flag becomes 1.

  The end_of_slice_flag indicates a flag indicating whether or not all macroblock syntax included in the slice has been transmitted. When 0, the end_of_slice_flag indicates that there is a macroblock syntax that has not yet been transmitted. On the other hand, a case of 1 indicates that all macroblock syntaxes in the slice have been transmitted.

  FIG. 14 is a diagram illustrating an example of the syntax of a macroblock layer for a stereo image subsampled in a lattice shape. First, the value of squeezing_flag is determined. If the squeezing flag is not squeezed, the macroblock contains two parallax images, so the macroblock type (information indicating whether the prediction is intraframe prediction or interframe prediction, etc.) Mb_type indicating 2 is multiplexed, and viewCnt is set to 2.

  On the other hand, when squeezed, the parallax image includes only either the left or the right, so one mb_type is multiplexed and viewCnt is set to 1.

  Next, the prediction mode information in the macroblock is multiplexed by the number of viewCnts. That is, when a macroblock is interframe prediction encoded and the inside of the macroblock is further divided into four submacroblocks (mb_type [viewIdx]! = INTRA && NumNumPart == 4), Sub_mb_pred (mb_type [viewIdx]) is multiplexed by the number of sub macroblocks. When the macroblock is subjected to intraframe prediction coding or when the number of sub macroblocks is not four, mb_pred (mb_type [viewIdx]) is multiplexed.

  FIG. 15 is a diagram illustrating an example of syntax indicating information related to a macroblock prediction mode. Inside this syntax, use_sameview_predmode_flag indicating whether to predict the prediction mode from an encoded block of the same parallax image is multiplexed. When this value is 0, it indicates that the prediction mode and motion vector of the target block are predictively encoded from the prediction mode and motion vector of the neighboring blocks used for the pixels of the same parallax image. On the other hand, when this value is 1, the prediction mode or motion vector of the target block is predicted from the prediction mode or motion vector for a block or peripheral block of a different parallax image at the same position before encoded sub-sample, for example. Indicates encoding.

  For prediction of prediction modes and motion vector prediction methods, see, for example, H.264. A method similar to that of H.264 may be used. For example, when mb_type [viewIdx] is INTRA_NxM indicating intra-frame prediction, prev_intraNxM_pred_mode_flag [lumaBlkIdx] is multiplexed by the number of internal intra prediction unit blocks.

  prev_intraNxM_pred_mode_flag [lumaBlkIdx] is the same prediction mode as the prediction mode of the left pixel block (hereinafter referred to as “left block”) belonging to the same encoded parallax image or the pixel block at the same position of a different parallax image. A flag that indicates whether to use the mode.

  When this flag is 1, that is, when the same prediction mode is not used, when the prediction mode of the left block is different from the prediction mode of the coding unit, the difference value is multiplexed as rem_intraNxM_pred_mode. Further, when the input image is a color signal, for example, YUV420, intra_chroma_pred indicating an intra-frame prediction mode for the color difference signal is separately transmitted.

  When there is no left pixel block belonging to the same parallax image, the prediction mode of the upper block is used, and when neither left nor upper is present, a predetermined prediction mode may be used.

  On the other hand, when mb_type indicates inter-frame prediction, the difference from the predicted motion vector of the motion vector of the corresponding sub-block is multiplexed by the number of internal sub-blocks. However, when the motion vector difference information is 0, the following information regarding the motion vector difference is not transmitted with the 0th macroblock type of the sub-macroblock as DIRECT (in the case of MbPartPredMode (mb_type, 0) == Direct) Equivalent to

  When the sub macroblock is not DIRECT, motion vector difference information is transmitted for each submacroblock. For example, H.M. In H.264, inter-frame prediction can be block-referenced from either or both of the two reference images (L0, L1). However, when the type of the sub macroblock is Pred_L1, that is, the prediction is not performed only from the reference image L1, the difference between the motion vector for the reference image L0 and the motion vector predicted from the neighboring blocks is multiplexed in the horizontal and vertical directions. (Mvd — 10 [mbPartIdx] [0] [compIdx]).

  Next, for example, when the type of the sub-macroblock is Pred_L0, that is, the prediction is not performed only from the reference image L0, the difference between the motion vector for the reference image L1 and the motion vector predicted from the neighboring blocks is multiplexed in both the horizontal and vertical directions. (Mvd_l1 [mbPartIdx] [0] [compIdx]).

  FIG. 16 is a diagram illustrating an example of syntax for sub macroblock prediction modes and motion vectors. First, macroblock types of four sub-blocks are transmitted in order, and then use_sameview_predmode_flag indicating whether prediction of prediction mode is predicted from an encoded block of the same disparity image is multiplexed for each of the four sub-blocks. Is done. Next, the syntax indicating the motion vector information for the subpartitions in the subblock is multiplexed for each subpartition. The same data structure as that used when multiplexing, for example, the motion vector information of FIG. 15 is multiplexed. This can be realized by using a structure.

  17 and 18 are flowcharts for explaining an image coding method according to the present embodiment. The process of FIG. 17 is executed by the image encoding device 20, for example.

  In step S101 of FIG. 17, the parallax image sampling position interpolator 1154 generates an interpolation pixel for prediction. This interpolated pixel becomes a prediction signal. The interpolation pixel is generated for each parallax image, and the interpolation pixel of the left eye image is generated from the pixel of the left eye image. Further, the interpolation pixel of the right eye image is generated from the pixel of the right eye image.

  In step S102, the intra-frame predictor 1151 or the inter-frame predictor 1152 generates a prediction residual that is a residual between the prediction signal and the pixel to be processed. Note that the interframe predictor 1152 generates a prediction residual after performing motion vector detection for motion compensation.

  In step S103, the orthogonal transformer 104 orthogonally transforms the prediction residual to generate a transform coefficient, the quantizer 106 quantizes the transform coefficient, and the entropy encoder 108 converts the quantized transform coefficient into an entropy code. Turn into. In step S103, the entropy encoder 108 further encodes accompanying information. The accompanying information includes prediction mode information and stereo image format information. In addition, when inter-frame prediction is performed, motion vector information is further encoded.

  In step S104, the inverse quantizer 109 inversely quantizes the quantized transform coefficient, and the inverse orthogonal transformer 110 performs inverse orthogonal transform on the inversely quantized transform coefficient. As a result, a decoded prediction residual signal is obtained.

  In step S105, the local decoded image signal generator 111 generates a local decoded image from the decoded prediction residual signal and pixels that have already been decoded. The locally decoded image has a predetermined stereo image format based on the stereo image format information. In step S105, the loop filter 113 may further perform filter processing on the locally decoded image.

  In step S106, the frame memory 114 stores the locally decoded image.

  FIG. 18 is a flowchart showing a part of the processing performed in step S103. FIG. 18 illustrates a process related to a prediction residual among the processes in step S103. In step S131 of FIG. 18, the orthogonal transformer 104 orthogonally transforms the prediction residual for each predetermined range in the picture. The predetermined range is, for example, a rectangle such as a block or a macro block. Thereby, orthogonal transform coefficient information 105 is generated.

  In step S132, the entropy encoder 108 converts the orthogonal transform coefficient information 105 generated in step S131 into an entropy code and multiplexes the encoded data. Prior to entropy coding, the orthogonal transform coefficient information 105 may be quantized by the quantizer 106. Also, motion vector information, prediction mode information, and stereo image format information may be multiplexed with the encoded data.

(decoder)
FIG. 19 is a diagram showing an image decoding apparatus according to the present embodiment. The image decoding apparatus 30 in FIG. 19 decodes input encoded data and outputs a decoded image. The encoded data processed by the image decoding device 30 is, for example, encoded data generated by the image encoding device 20.

  The image decoding apparatus 30 includes an entropy decoder 200, an inverse quantizer 109, an inverse orthogonal transformer 110, a local decoded image signal generator 202, a loop filter 113, a frame memory 114, and a predicted image generator 204. .

  The entropy decoder 200 decodes the encoded data 117 in the reverse procedure of entropy encoding. Thereby, quantized orthogonal transform coefficient information 107, information 116 including motion vector information, prediction mode information, and stereo image format information, and orthogonal transform information are obtained. Based on the orthogonal transform information, the inverse quantizer 109 and the inverse orthogonal transformer 110 sequentially perform inverse processing that is paired with the processing of the quantizer 106 and the orthogonal transformer 104 on the quantized orthogonal transform coefficient information 107. A residual signal 201 is generated.

  The prediction image creator 204 receives information 116 including motion vector information, prediction mode information, and stereo image format information. The predicted image generator 204 generates a predicted image signal 102 based on the decoded image signal 203 and information 116 including motion vector information, prediction mode information, and stereo image format information.

  The local decoded image signal generator 202 adds the residual signal 201 and the predicted image signal 102. The added signal is output as a decoded image signal 203 and stored in the frame memory 114. The added signal may be subjected to filter processing similar to that of the loop filter 113 in the encoder by the loop filter 113. The decoded image signal 203 stored in the frame memory 114 is output after being rearranged in the display order.

  FIG. 20 is a diagram illustrating an example of a detailed functional configuration of the predicted image creator 204. The predicted image generator 204 generates a predicted image signal 102 for each parallax image from the input decoded image signal 203. The predicted image creator 204 includes a parallax image separator 2034, a parallax image sampling position interpolator 2033, a switcher 2032, an intra-frame predictor 2030, and an inter-frame predictor 2031.

  The parallax image separator 2034 separates the locally decoded image signal 112 into two parallax images. For example, the parallax image separator 2034 performs separation into parallax images based on information related to a stereo image format multiplexed in the encoded data 117 with the syntax illustrated in FIG. 11. The parallax image separator 2034 further cancels the squeezing of the squeezed image.

  The parallax image sampling position interpolator 2033 interpolates pixels at pixel positions that are not sampled for each parallax image. The parallax image sampling position interpolator 2033 performs pixel interpolation based on information related to the stereo image format multiplexed on the encoded data 117 with the syntax shown in FIG. The interpolated parallax image is input to the intra-frame predictor 2030 or the inter-frame predictor 2031 by the switcher 2032.

  For example, the switching unit 2032 and the intra-frame prediction unit 2031 perform inter-frame prediction based on the prediction mode information multiplexed on the encoded data 117 with the syntax shown in FIGS. 13 to 16 and information on the stereo image format. One of the units 2032 is selected.

  The intra-frame predictor 2030 performs intra-frame prediction using, as a reference image, the local decoded image signal 112 that is separated for each parallax image or is squeezed and interpolated with pixels at pixel positions that have not been sampled. The intra-frame prediction performed by the intra-frame predictor 2030 is the same as the intra-frame predictor 1151 of the image encoding device 20. As a result, a predicted image signal 102 based on intra-frame prediction is created.

  The inter-frame predictor 2031 performs motion compensation on the reproduced image signal in the frame memory using, for example, a motion vector multiplexed on the encoded data 117 with the syntax of FIG. 14 or FIG. 15, and is based on inter-frame prediction. Create a predicted image signal. The inter-frame prediction process by motion compensation of the inter-frame predictor 2031 is the same process as the inter-frame predictor 1152 of the image encoding device 20.

  21 and 22 are flowcharts for explaining the image decoding method according to the present embodiment. The process of FIG. 21 is executed by the image decoding device 30, for example.

  In step S201 of FIG. 21, the entropy decoder 200 entropy-decodes the encoded data 117, and information 116 including quantized orthogonal transform coefficient information 107, motion vector information, prediction mode information, and stereo image format information, And orthogonal transformation information is obtained.

  In step S201, the inverse quantizer 109 further inversely quantizes the quantized orthogonal transform coefficient information 107 to generate orthogonal transform coefficient information, and the inverse orthogonal transformer 110 performs inverse orthogonal transform on the orthogonal transform coefficient information. The residual signal 201 is output. In step S202, the predicted image creator 204 generates the predicted signal 102 from the decoded image signal 203 in accordance with the prediction mode information.

  In step S <b> 203, the local decoded image signal generator 202 generates a decoded image signal 203 from the prediction signal 102 and the residual signal 201. In step S <b> 203, the decoded image signal 203 may be further filtered by the loop filter 113.

  In step S <b> 204, the decoded image signal 203 is stored in the frame memory 114. The decoded image signal 203 stored in the frame memory 114 is output for each frame in the display order.

  FIG. 22 is a flowchart showing a part of the processing performed in step S201. In step S211 of FIG. 22, the entropy decoder 200 decodes the encoded data 117 to obtain a transform coefficient of the prediction residual. If this transform coefficient is quantized, in step S211, the inverse quantizer 109 further performs inverse quantization. Thereby, orthogonal transformation coefficient information is output.

  In step S212, the inverse orthogonal transformer 110 performs inverse orthogonal transform on the orthogonal transform coefficient information, and outputs a residual signal 201.

  According to the above embodiments, the stereo image subsampled and merged in a grid pattern is subjected to prediction processing, filtering processing, and orthogonal transformation processing separately for each parallax image, thereby reducing the influence of parallax and phase shift due to squeeze. Effective excluded prediction, filtering, and orthogonal transformation can be performed, and encoding efficiency can be improved.

(Realization by computer etc.)
Note that the stereo image encoding device and the stereo image decoding device according to the present embodiment may be realized by a personal computer (PC), for example. Further, in the stereo image encoding device and the stereo image decoding device according to the embodiment of the present invention, for example, the CPU uses a main memory such as a RAM as a work area according to a program stored in a ROM, a hard disk device, or the like, Executed.

  It should be noted that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  As described above, the stereo image encoding device according to the present invention is useful when transmitting a stereo image through an existing transmission channel.

20 Image coding apparatus 30 Image decoding apparatus 100 Input image signal 101 Difference signal generator 102 Predicted image signal 103 Prediction error signal 104 Orthogonal transformer 105 Orthogonal transform coefficient information 106 Quantizer 107 Quantized orthogonal transform coefficient information 108 Entropy code 109 Inverse quantizer 110 Inverse orthogonal transformer 111 Local decoded image signal generator 112 Local decoded image signal 113 Loop filter 114 Frame memory 115 Predictive image generator 116 Information 117 Encoded data 118 Input frame buffer 200 Entropy decoder 201 Residual signal 202 Local decoded image signal generator 203 Decoded image signal 204 Predicted image generator 1150 Mode determiner 1151, 2030 Intraframe predictor 1152, 2031 Interframe predictor 1153 Motion vector detector 1154, 2 33 parallax images sampling position interpolator 1155,2034 parallax image separator 2032 switcher

Claims (8)

A first stereo image signal having a first parallax image having a pixel at a first phase position of the checkerboard pattern and a second parallax image having a pixel at a second phase position of the checkerboard pattern. (A) prediction residual for each of the parallax images, and (B) prediction mode for each of the parallax images. A decoding step of decoding information and (C) joint information indicating an arrangement of the first parallax image and the second parallax image on the picture;
The parallax image according to the prediction mode information with reference to each pixel value on the already decoded parallax image and pixel values interpolated from surrounding pixels at positions where no pixel exists on the already decoded parallax image A prediction signal generation step for generating a prediction signal for each
A decoded image generating step of generating a decoded image signal by adding the prediction signal and the prediction residual according to the combination information;
A stereo image decoding method comprising: In the prediction signal generation step, the prediction signal is generated for each predetermined range in the picture,
2. The decoding step according to claim 1, wherein, in each of the predetermined ranges, an index calculated based on information relating to the prediction mode in the predetermined range that has already been decoded is decoded. Stereo image decoding method. In the decoding step, for each of the parallax images, further, information on a motion vector for a reference image having the same parallax in another picture is decoded,
The information related to the motion vector is information related to a difference between the motion vector and a motion vector in a predetermined range that has already been decoded,
3. The stereo image decoding method according to claim 2, wherein in the prediction signal generation step, the prediction signal in the reference image is generated, and motion compensation is performed using the motion vector. The encoded data is obtained by orthogonally transforming the prediction residual for each predetermined range of an image obtained by interpolating a pixel at an unextracted pixel position from the pixel of the parallax image for each parallax image. When including the sign of the coefficient,
The decoding step includes
For each parallax image, a prediction residual is obtained by inverse orthogonal transformation with respect to the transform coefficient for each predetermined range, and the prediction residual of the pixel position of the parallax image is sampled among the prediction residuals. The stereo image decoding method according to claim 3, further comprising an inverse orthogonal transform step of outputting. The decoding step includes
A transform coefficient decoding step for decoding a transform coefficient of each of the prediction residuals of a predetermined range composed of pixels belonging to even lines and a predetermined range composed of pixels belonging to odd lines for each of the parallax images;
An inverse orthogonal transform step for generating a prediction residual by performing an inverse orthogonal transform on each of a predetermined range including pixels belonging to the even lines and a predetermined range including pixels belonging to the odd lines;
5. The stereo image decoding method according to claim 4, further comprising: In the decoded image signal, pixels included in the two parallax images are arranged in a predetermined arrangement order for each picture based on the combination information.
The stereo image decoding method according to claim 5, further comprising a filter processing step for performing a filter process for each pixel having the same pixel position phase with respect to the pixels included in the decoded image signal. In the decoded image signal, when the first parallax image and the second parallax image are arranged in a complementary lattice shape,
The stereo image decoding method according to claim 6, wherein in the filter processing step, filter processing using every other pixel of the decoded image signal is performed. A first stereo image signal having a first parallax image having a pixel at a first phase position of the checkerboard pattern and a second parallax image having a pixel at a second phase position of the checkerboard pattern. A prediction signal of each parallax image of a picture in which the parallax image and the second parallax image are combined, according to a prediction mode selected from a plurality of prediction modes, A prediction signal generation step of generating by referring to a pixel value interpolated from neighboring pixels at positions where no pixel exists on the already encoded parallax image;
A residual generation step for generating a prediction residual between each parallax image and the prediction signal;
(A) the prediction residual for each parallax image, (B) information on the selected prediction mode for each parallax image, and (C) the first parallax image and the second parallax image on the picture An encoding step for encoding combined information indicating the arrangement of
A decoding step of decoding the prediction residual to generate a decoded prediction residual;
A decoded image generating step of generating a decoded image signal by adding the decoded prediction residual to the prediction signal based on the information related to the prediction mode and the combined information;
A stereo image encoding method comprising:

Images