WO2012128242A1 - 画像処理装置、画像処理方法、及び、プログラム - Google Patents
画像処理装置、画像処理方法、及び、プログラム Download PDFInfo
- Publication number
- WO2012128242A1 WO2012128242A1 PCT/JP2012/056975 JP2012056975W WO2012128242A1 WO 2012128242 A1 WO2012128242 A1 WO 2012128242A1 JP 2012056975 W JP2012056975 W JP 2012056975W WO 2012128242 A1 WO2012128242 A1 WO 2012128242A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- prediction
- image
- unit
- vector
- occlusion
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/513—Processing of motion vectors
- H04N19/517—Processing of motion vectors by encoding
- H04N19/52—Processing of motion vectors by encoding by predictive encoding
Definitions
- the present technology relates to an image processing device, an image processing method, and a program, for example, to obtain a vector with high prediction accuracy as a prediction vector such as a parallax vector representing a parallax of an image of a certain viewpoint with respect to an image of another viewpoint.
- the present invention relates to an image processing apparatus, an image processing method, and a program.
- an encoding method for encoding an image of a plurality of viewpoints such as a 3D (Dimension) image
- an MVC (Multiview Video Coding) method which is an extension of the AVC (Advanced Video Coding) (H.264 / AVC) method, etc. is there.
- an image to be encoded is a color image having a value corresponding to light from a subject as a pixel value, and each of the color images of a plurality of viewpoints is a color image of the viewpoint as necessary.
- encoding is performed with reference to color images of other viewpoints.
- one viewpoint color image among a plurality of viewpoint color images is used as a base view (Base View) image, and other viewpoint color images are used as dependent views (Dependent View). ) Image.
- the color image of the base view is encoded with reference to only the image of the base view, and the color image of the dependent view requires the image of another view in addition to the image of the dependent view. And is encoded according to the reference.
- parallax prediction for generating a predicted image with reference to the color image of another view is performed as necessary, and is encoded using the predicted image.
- a certain viewpoint # 1 is a base view
- another viewpoint # 2 is a dependent view.
- the parallax prediction is performed on the color image of the viewpoint # 2 with reference to the color image of the viewpoint # 1, and the prediction image obtained by the parallax prediction is encoded (predictive coding)
- predictive coding For example, a disparity vector representing a disparity with respect to the color image of the viewpoint # 1 of the target block that is a macroblock having 16 ⁇ 16 pixels in the horizontal and vertical directions is detected.
- a prediction vector obtained by predicting the disparity vector of the target block is obtained, and a residual vector that is a difference between the disparity vector and the prediction vector is encoded.
- the residual vector code amount tends to increase as the residual vector increases. Therefore, if the residual vector is small, that is, if the prediction vector has high prediction accuracy (prediction vector). The more similar to the disparity vector, the better the coding efficiency.
- a parallax information image having, as a pixel value, parallax information related to the parallax for each pixel of a color image of each viewpoint, in addition to the color image of each viewpoint, is adopted as a plurality of viewpoint images.
- standards such as the MPEG3DV method are being formulated as an encoding method for encoding the color image of each color and the parallax information image of each viewpoint.
- each viewpoint color image and each viewpoint parallax information image are encoded in the same manner as in the MVC format.
- a prediction vector (for a disparity vector) of the target block is obtained from a disparity vector for a block around the target block.
- a prediction vector for the disparity vector is also obtained.
- a method has been proposed (see, for example, Non-Patent Document 1).
- the parallax information image of the viewpoint # 2 that is the dependent view may have an occlusion part that is a part having no corresponding point in the parallax information image of the viewpoint # 1 that is the base view due to the influence of the parallax. .
- the MVC method may require a prediction vector with poor prediction accuracy.
- the present technology has been made in view of such a situation, and makes it possible to obtain a vector with high prediction accuracy as a prediction vector.
- the image processing apparatus or the program according to one aspect of the present technology may include another picture different from the picture of the target block of the target block to be processed of the depth image having depth information regarding the parallax for each pixel of the color image as a pixel value.
- a prediction vector generation unit that generates a prediction vector of a shift vector representing a shift with respect to an image by a different vector prediction method depending on whether the target block is an occlusion block including an occlusion portion in which no corresponding point exists in the other picture.
- An image processing method represents a shift of a target block to be processed of a depth image having depth information relating to a parallax for each pixel of a color image as a pixel value with respect to another picture different from the picture of the target block.
- the image processing method includes a step of generating a prediction vector of a shift vector by a different vector prediction method depending on whether the target block is an occlusion block including an occlusion portion in which no corresponding point exists in the other picture.
- a prediction vector is generated by a different vector prediction method depending on whether the target block is an occlusion block including an occlusion portion in which no corresponding point exists in the other picture.
- the image processing apparatus may be an independent apparatus or an internal block constituting one apparatus.
- the program can be provided by being transmitted through a transmission medium or by being recorded on a recording medium.
- a prediction vector with high prediction accuracy can be obtained.
- FIG. 2 is a block diagram illustrating a configuration example of an encoder 11.
- FIG. It is a figure explaining the macroblock type of a MVC (AVC) system. It is a figure explaining the prediction vector of a MVC (AVC) system. It is a figure explaining the prediction vector of a MVC (AVC) system.
- 3 is a block diagram illustrating a configuration example of an encoder 22.
- FIG. 4 is a block diagram illustrating a configuration example of a disparity prediction unit 234.
- FIG. 12 is a flowchart for describing an encoding process for encoding a parallax image D # 2 of a viewpoint # 2. It is a flowchart explaining an occlusion detection process. It is a figure explaining an occlusion detection process. It is a flowchart explaining a parallax prediction process. It is a flowchart explaining a prediction vector production
- 11 is a block diagram illustrating a configuration example of a decoder 311. FIG. 11 is a block diagram illustrating a configuration example of a decoder 322. FIG.
- FIG. 16 is a flowchart for describing a decoding process for decoding encoded data of a parallax image D # 2 of viewpoint # 2. It is a flowchart explaining a parallax prediction process.
- 6 is a block diagram illustrating another configuration example of the encoder 22.
- FIG. 5 is a block diagram illustrating a configuration example of a disparity prediction unit 534.
- FIG. It is a flowchart explaining a parallax prediction process. It is a flowchart explaining a prediction vector production
- FIG. 16 is a flowchart for describing a decoding process for decoding encoded data of a parallax image D # 2 of viewpoint # 2. It is a flowchart explaining a parallax prediction process.
- 6 is a block diagram illustrating another configuration example of the encoder 22.
- FIG. 5
- FIG. 16 is a flowchart for describing a decoding process for decoding encoded data of a parallax image D # 2 of viewpoint # 2. It is a flowchart explaining a parallax prediction process. It is a flowchart explaining a prediction vector production
- FIG. 18 is a block diagram illustrating a configuration example of an embodiment of a computer to which the present technology is applied. It is a figure which shows the schematic structural example of the television apparatus to which this technique is applied. It is a figure which shows the schematic structural example of the mobile telephone to which this technique is applied. It is a figure which shows the schematic structural example of the recording / reproducing apparatus to which this technique is applied. It is a figure which shows the schematic structural example of the imaging device to which this technique is applied.
- FIG. 41 is a diagram illustrating parallax and depth.
- the depth of the subject M from the camera c1 (camera c2).
- the depth Z that is the distance in the direction is defined by the following equation (a).
- L is a horizontal distance between the position C1 and the position C2 (hereinafter, referred to as an inter-camera distance).
- D is the position of the subject M on the color image photographed by the camera c2 from the horizontal distance u1 of the position of the subject M on the color image photographed by the camera c1 from the center of the color image.
- f is the focal length of the camera c1, and in the formula (a), the focal lengths of the camera c1 and the camera c2 are the same.
- the parallax d and the depth Z can be uniquely converted. Therefore, in this specification, the image representing the parallax d and the image representing the depth Z of the two viewpoint color images captured by the camera c1 and the camera c2 are collectively referred to as a depth image (parallax information image).
- the depth image may be an image representing the parallax d or the depth Z
- the pixel value of the depth image is not the parallax d or the depth Z itself but the parallax d as a normal value.
- the normalized value the value obtained by normalizing the reciprocal 1 / Z of the depth Z, and the like can be employed.
- the value I obtained by normalizing the parallax d with 8 bits (0 to 255) can be obtained by the following equation (b). Note that the normalization bit number of the parallax d is not limited to 8 bits, and other bit numbers such as 10 bits and 12 bits may be used.
- D max is the maximum value of the parallax d
- D min is the minimum value of the parallax d.
- the maximum value D max and the minimum value D min may be set in units of one screen, or may be set in units of a plurality of screens.
- the value y obtained by normalizing the reciprocal 1 / Z of the depth Z by 8 bits (0 to 255) can be obtained by the following equation (c).
- the normalized bit number of the inverse 1 / Z of the depth Z is not limited to 8 bits, and other bit numbers such as 10 bits and 12 bits may be used.
- Z far is the maximum value of the depth Z
- Z near is the minimum value of the depth Z.
- the maximum value Z far and the minimum value Z near may be set in units of one screen or may be set in units of a plurality of screens.
- an image having a pixel value of the value I obtained by normalizing the parallax d, and an inverse 1 / of the depth Z is collectively referred to as a depth image (parallax information image).
- a depth image parllax information image
- the color format of the depth image is YUV420 or YUV400, but other color formats are also possible.
- the value I or the value y is set as the depth information (disparity information). Further, a map obtained by mapping the value I or the value y is a depth map (disparity map).
- FIG. 1 is a diagram for explaining a method for obtaining a prediction vector of a disparity vector of a disparity information image by the MVC method.
- Non-Patent Document 1 describes obtaining a prediction vector of a disparity vector of a disparity information image by the MVC method.
- color image C # 1 of viewpoint # 1 which is a color image of two viewpoints
- color image C # 2 of viewpoint # 2 different from viewpoint # 1 and two viewpoints It is assumed that there is a parallax information image D # 1 for viewpoint # 1 and a parallax information image D # 2 for viewpoint # 2, which are parallax information images.
- the parallax information image D # 1 (picture) of the viewpoint # 1 among the parallax information image D # 1 of the viewpoint # 1 and the parallax information image D # 2 of the viewpoint # 2 is used as a reference picture. Assume that a prediction vector of a disparity vector used in disparity prediction for generating a prediction image of the disparity information image D # 2 of the viewpoint # 2 is obtained.
- a block (macroblock) X having a t-th picture that is a t-th (time t) picture from the top of the disparity information image D # 2 of the viewpoint # 2 is a target block to be encoded (or decoded).
- a picture having a target block of the disparity information image D # 2, that is, a picture to be encoded is also referred to as a target picture.
- the prediction vector (of the disparity vector) of the target block X of the disparity information image D # 2 is (in raster scan order) of the blocks around the target block X (hereinafter also referred to as peripheral blocks). It is obtained from the disparity vector of an already encoded (decoded) block.
- a prediction vector is obtained by a vector prediction method (hereinafter also referred to as a median prediction method or a median predictor) that obtains (median value) as a prediction vector of the target block X.
- a vector prediction method hereinafter also referred to as a median prediction method or a median predictor
- the median as the prediction vector of the target block X is calculated independently for each of the x component and the y component.
- the disparity information images D # 1 and D # 2 include a background (disparity information thereof) and a rectangular foreground (disparity information thereof) as an object existing on the near side of the background. Suppose that it is reflected.
- the color images C # 1 and C # 2 have two cameras arranged on the same horizontal plane in a direction orthogonal to a straight line connecting the positions of the two cameras. It is assumed that the color image is obtained by shooting.
- the color image C # 1 of the viewpoint # 1 is a color image captured by the right camera of the two cameras toward the subject (capture direction), and the color image C # 2 of the viewpoint # 2 Is a color image taken by the left camera of the two cameras.
- the foreground is reflected in the center of the parallax information image D # 2 of the viewpoint # 2
- the foreground is the parallax in the parallax information image D # 1 of the viewpoint # 1, as shown in FIG. Due to the effect, it appears slightly to the left of the center.
- the disparity information image D # 2 of the viewpoint # 2 includes a part of the background (the part hatched in FIG. 1) that appears on the left side of the foreground that is reflected in the disparity information image D # 2. ) Is an occlusion portion where no corresponding point exists in the parallax information image D # 1 of the viewpoint # 1.
- a block (macro block) including an occlusion part is also referred to as an occlusion block
- a block other than the occlusion block is also referred to as a non-occlusion block.
- the color image C # 1 of the viewpoint # 1 is a color image taken by the right camera of the two cameras
- the color image C # 2 of the viewpoint # 2 is the one of the two cameras. If the viewpoint # 1 is positioned on the right side and the viewpoint # 2 is positioned on the left side, the parallax information image D # 2 of the viewpoint # 2 is the left side of the foreground.
- the portion of the background that touches is the occlusion portion. Therefore, the left side of the occlusion portion is the background, and the right side is the foreground.
- the pixel value of the foreground is almost the same value unless the foreground has large unevenness in the depth direction.
- the background pixel values are substantially the same.
- the foreground and the background generally have relatively different positions in the depth direction (because the foreground is located on the near side and the background is located on the far side), so that the pixel values are also relatively different.
- the foreground pixel values are substantially the same, and the background pixel values are also substantially the same.
- a disparity vector as a displacement vector representing a displacement of a block of the disparity information image D # 2 with respect to the disparity information image D # 1 that is a reference picture is used.
- ME Motion Estimation
- the background in the occlusion block does not appear in the same position as the occlusion block in the disparity image information D # 1 that is the reference picture (the foreground is The disparity vector is not a zero vector.
- the disparity vector of the target block X matches the disparity vector of the neighboring block A adjacent above.
- the disparity vector of the peripheral block B adjacent to the left and the peripheral block C adjacent to the upper right is significantly different (correlation is small).
- the peripheral block B adjacent to the left of the occlusion block since the median of the disparity vector of the peripheral block around the target block is obtained as the prediction vector of the target block (disparity vector thereof), the peripheral block B adjacent to the left of the occlusion block, and If the disparity vector of the neighboring block C adjacent to the upper right is significantly different from the disparity vector of the occlusion block, a vector that is significantly different from the disparity vector is obtained as the prediction vector of the disparity vector of the occlusion block. Become.
- the prediction vector of the disparity vector of the occlusion block is significantly different from the disparity vector, that is, if the prediction accuracy of the prediction vector is poor, the residual vector, which is the difference between the disparity vector and the prediction vector, becomes large. In this case, the coding amount of the residual vector increases, and the coding efficiency deteriorates.
- the occlusion part it is desirable to adopt a vector prediction method different from the MVC method so that a prediction vector with good prediction accuracy can be generated.
- the picture of the disparity information image D # 1 of the first viewpoint # 1 is extracted from the target block of the disparity information image D # 2 of the second viewpoint # 2.
- a proximity prediction method is employed in which a vector to a proximity portion close to the outside of the occlusion corresponding portion at the same position as the occlusion portion is generated as a prediction vector of the target block.
- FIG. 2 is a diagram for explaining the outline of the present technology.
- the viewpoint # 1 is located on the right side and the viewpoint # 2 is located on the left side.
- the portion of the background that touches the left side of the foreground is an occlusion part. Therefore, the left side of the occlusion part is the background, and the right side is the foreground.
- the disparity vector of the target block to be processed in the disparity information image D # 2 of the viewpoint # 2 is referred to the disparity information image D # 1 (the picture) of the viewpoint # 1, which is another picture different from the target picture
- the prediction vector of the target block is an occlusion block that includes an occlusion part in which the corresponding block does not exist in the disparity information image D # 1 of the viewpoint # 1 as the reference picture.
- the prediction vector of the target block is an occlusion block that includes an occlusion part in which the corresponding block does not exist in the disparity information image D # 1 of the viewpoint # 1 as the reference picture.
- the occlusion portion of the parallax information image D # 2 of the viewpoint # 2 is detected.
- the detection of the occlusion part is obtained by warping the parallax information image D # 1 of the viewpoint # 1 that is a reference picture, and the warped parallax information image obtained by converting the parallax information image D # 1 into an image obtained by the viewpoint # 2 D '# 1 is used.
- each pixel (value) of the parallax information image D # 1 corresponds to the parallax between the viewpoints # 1 and # 2 in each pixel.
- a warped parallax information image (warped depth image) D ′ # 1 is generated.
- the portion that is shown in the parallax information image D # 2 but not in the parallax information image D # 1 has no pixel value as shown in FIG. In other words, it becomes a perforated state.
- a portion having no pixel value in the warped parallax information image D ′ # 1 (hereinafter also referred to as a portion having no pixel value) is detected, and a portion in the same position as the portion having no pixel value in the parallax information image D # 2 is detected.
- the occlusion portion can be detected.
- the target block is an occlusion block including an occlusion part
- an occlusion correspondence part at the same position as the occlusion part of the disparity information image D # 1 of the first viewpoint # 1 that is a reference picture from the target block.
- a prediction vector of the target block is generated by a proximity prediction method that generates a vector to a proximity portion close to the background on the outside of the target block as a prediction vector of the target block.
- the start of the occlusion portion in the block is started.
- a pixel S and a finished end pixel E are detected.
- FIG. 2 in a target picture of the disparity information image D # 2, three blocks (macroblocks) B # 1, B # 2, and B # arranged in the horizontal direction (from left to right) (x direction) are arranged. 3 is an occlusion block including an occlusion part.
- the start pixel S # 1 of the occlusion portion of the block B # 1 is the eighth pixel (from the left of the block), and the end pixel E # 1 is the sixteenth pixel.
- start pixel S # 2 is the first pixel
- end pixel E # 2 is the 16th pixel.
- start pixel S # 3 is the first pixel
- end pixel E # 3 is the twelfth pixel.
- the prediction vector PMV # i (of x Component).
- the prediction vector PMV # (i-1) of the occlusion block B # (i-1) adjacent to the left of the target block B # i it ends from the start pixel S # i of the occlusion block
- the prediction vector PMV # i of the target block B # i is obtained.
- the prediction of the block B # (i-1) is a 0 vector.
- Equation (1) represents the x component of the prediction vector PMV # i.
- a vector from the target block B # i to the background that is a close part in contact with the outside of the left boundary of the occlusion corresponding part of the parallax information image D # 1 is Generated as a prediction vector PMV # i.
- FIG. 3 is a diagram for explaining a method for generating a prediction vector of a target block in the proximity prediction method.
- the viewpoint # 1 is located on the right side
- the viewpoint # 2 is located on the left side
- the background touching the left side of the foreground in the parallax information image D # 2 of the viewpoint # 2 Shows the method of generating the prediction vector of the occlusion block in the proximity prediction method when the part is the occlusion part, that is, the left side of the occlusion part is the background and the right side is the foreground. Yes.
- FIG. 3 a part of the region R # 2 including the occlusion part of the parallax information image D # 2 is enlarged, and the region R # of the parallax information image D # 1 at the same position as the region R # 2 is enlarged. 1 is shown enlarged.
- FIG. 3 in the target picture of the disparity information image D # 2, three blocks B # 1, B # 2, and B # 3 arranged in the horizontal direction are occlusion blocks including an occlusion portion.
- the start pixel and the end pixel for the block B # i are represented by S # i and E # i, respectively.
- the viewpoint # 1 When the viewpoint # 1 is located on the right side, the viewpoint # 2 is located on the left side, and in the parallax information image D # 2 of the viewpoint # 2, the background part on the left side of the foreground is an occlusion part.
- the prediction vector PMV # i of the block B # i that is an occlusion block is obtained in the raster scan order that is the encoding order (decoding order) according to the equation (1).
- Each of the prediction vectors PMV # i of the block B # i that is an occlusion block is a background that is a proximity part that touches the outside of the left boundary of the occlusion corresponding part of the disparity information image D # 1 from the target block B # i That is, in the parallax information image D # 1, this is a vector to the background portion that touches the left of the foreground.
- FIG. 4 is a diagram for explaining a method for generating a prediction vector of a target block in the proximity prediction method.
- the viewpoint # 1 is located on the left side and the viewpoint # 2 is located on the right side.
- the background part on the right side of the foreground is In the case of the occlusion part, that is, the right side of the occlusion part is the background and the left side is the foreground, a method for generating a prediction vector of the occlusion block in the proximity prediction method is shown.
- a partial region R # 2 including the occlusion portion of the parallax information image D # 2 is enlarged and the region R # 2 of the parallax information image D # 1 is enlarged.
- FIG. 4 as in FIG. 3, in the target picture of the disparity information image D # 2, three blocks B # 1, B # 2, and B # 3 arranged in the horizontal direction have occlusion portions.
- the start pixel and the end pixel of the block B # i are represented by S # i and E # i, respectively.
- the start pixel S # (i ⁇ 1) of the occlusion block B # (i ⁇ 1) adjacent to the left of the target block B # i.
- the prediction vector PMV # i of the target block B # i is obtained by subtracting the number of pixels E # (i-1) -S # (i-1) +1 from 1) to the end pixel E # (i-1). It is done.
- Equation (2) if the block B # (i-1) adjacent to the left of the target block B # i is not an occlusion block, as long as the calculation of Equation (2) is concerned, the prediction of that block B # (i-1)
- the vector PMV # (i-1) is the sum ⁇ (E #) of the number of pixels from the start pixel S # i to the end pixel E # i of the occlusion block B # i arranged continuously in the horizontal direction with the target block at the top. iS # i + 1).
- the number of occlusion blocks B # i arranged in a row in the horizontal direction starting from the target block is N, and the start pixel and end of the i-th occlusion block B # i from the beginning of the N occlusion blocks
- the summation ⁇ (E # iS # i + 1) ⁇ is the summation by changing i to an integer value from 1 to N.
- Equation (2) the vector from the target block B # i to the background that is a close part that contacts the outside of the right boundary of the occlusion corresponding part of the disparity information image D # 1 is Generated as a prediction vector PMV # i.
- the prediction vector PMV # i of the block B # i which is an occlusion block is obtained in the raster scan order which is the encoding order (decoding order) according to the equation (2).
- the prediction vector PMV # i of the block B # i which is an occlusion block, is a background part that is in close contact with the outside of the right boundary of the occlusion corresponding part of the disparity information image D # 1 from the target block B # i That is, in the parallax information image D # 1, this is a vector to the background portion that touches the right of the foreground.
- the occlusion block of the disparity information image D # 2 indicates a background that appears from the occlusion block to the outside of the occlusion corresponding portion of the disparity image information D # 1 at the same position as the occlusion portion.
- a simple vector is detected as a disparity vector.
- the vector from the target block B # i to the background that is a proximity part that contacts the outside of the boundary of the occlusion correspondence part of the parallax information image D # 1 Is generated as the prediction vector PMV # i of the target block B # i.
- a vector with high prediction accuracy that is, a vector similar to a disparity vector can be obtained as a prediction vector of an occlusion block.
- a residual vector becomes small and encoding is performed. Efficiency can be improved.
- the pixel value (disparity information) of the occlusion part of the disparity information image D # 2 is a background image that touches the occlusion corresponding part at the same position as the occlusion part of the disparity information image D # 1 that is the reference picture.
- a prediction vector with high prediction accuracy is obtained on the premise that the pixel value (disparity information) is substantially the same value, but this assumption is generally satisfied for a disparity information image.
- FIG. 5 is a block diagram illustrating a configuration example of an embodiment of a multi-view image encoder to which the present technology is applied.
- the multi-viewpoint image encoder in FIG. 5 is an encoder that encodes images of a plurality of viewpoints using, for example, the MVC method, and description of processing similar to that in the MVC method will be omitted as appropriate.
- multi-viewpoint image encoder is not limited to an encoder that uses the MVC method.
- the color image C # 1 of the viewpoint # 1 which is the color image of the two viewpoints # 1 and # 2
- the color image C # 2 of the viewpoint # 2 and
- the parallax information image D # 1 of the viewpoint # 1 and the parallax information image D # 2 of the viewpoint # 2 that are the parallax information images of the two viewpoints # 1 and # 2 are adopted.
- the color image C # 1 of the viewpoint # 1 and the parallax information image D # 1 are set as base view images, and the color image C # 2 and the parallax information image D # 2 of the remaining viewpoint # 2 are used. Are treated as dependent view images.
- a color image of three or more viewpoints and a parallax information image can be adopted, and among the color images of the three or more viewpoints and the parallax information image, Any one viewpoint color image and disparity information image can be used as a base view image, and the remaining viewpoint color images and disparity information images can be handled as dependent view images.
- the multi-view image encoder includes encoders 11, 12, 21, 22, DPB 31, and a multiplexing unit 32.
- the encoder 11 is supplied with the color image C # 1 of view # 1 and parallax related information (depth related information).
- the disparity related information is metadata of disparity information, and details thereof will be described later.
- the encoder 11 encodes the color image C # 1 of the viewpoint (view) # 1 using the disparity related information as necessary, and the encoded data of the color image C # 1 of the viewpoint # 1 obtained as a result is encoded. To the multiplexing unit 32.
- the encoder 12 is supplied with the color image C # 2 of viewpoint # 2 and parallax related information.
- the encoder 12 encodes the color image C # 2 of the viewpoint # 2 using the disparity related information as necessary, and multiplexes the encoded data of the color image C # 2 of the viewpoint # 2 obtained as a result To the unit 32.
- the encoder 21 is supplied with the parallax information image D # 1 of the viewpoint # 1 and the parallax related information.
- the encoder 21 encodes the parallax information image D # 1 of the viewpoint # 1 as necessary using the parallax related information, and the encoded data of the parallax information image D # 1 of the viewpoint # 1 obtained as a result is The data is supplied to the multiplexing unit 32.
- the encoder 22 is supplied with the parallax information image D # 2 of the viewpoint # 2 and the parallax related information.
- the encoder 22 encodes the disparity information image D # 2 of the viewpoint # 2 using the disparity related information as necessary, and the encoded data of the disparity information image D # 2 of the viewpoint # 2 obtained as a result is The data is supplied to the multiplexing unit 32.
- the DPB 31 refers to an image (decoded image) after local decoding obtained by encoding and locally decoding an image to be encoded at each of the encoders 11, 12, 21, and 22 when generating a predicted image. Temporarily store as a reference picture (candidate).
- the encoders 11, 12, 21, and 22 perform predictive encoding on the encoding target image. Therefore, the encoders 11, 12, 21, and 22 generate a predicted image to be used for predictive encoding, encode an encoding target image, perform local decoding, and obtain a decoded image.
- the DPB 31 is a shared buffer that temporarily stores decoded images obtained by the encoders 11, 12, 21, and 22.
- the encoders 11, 12, 21, and 22 are each decoded by the DPB 31.
- a reference picture to be referred to for encoding an image to be encoded is selected from the images.
- Each of the encoders 11, 12, 21, and 22 generates a prediction image using the reference picture, and performs image encoding (prediction encoding) using the prediction image.
- each of the encoders 11, 12, 21, and 22 is decoded by other encoders in addition to the decoded image obtained by itself. Images can also be referenced.
- the multiplexing unit 32 multiplexes the encoded data from each of the encoders 11, 12, 21, and 22, and outputs the multiplexed data obtained as a result.
- the multiplexed data output from the multiplexing unit 32 is recorded on a recording medium (not shown) or transmitted via a transmission medium (not shown).
- FIG. 6 is a block diagram illustrating a configuration example of a multi-view image generation apparatus that generates images of a plurality of viewpoints to be encoded in the multi-view image encoder of FIG.
- two cameras 41 and 42 are installed at positions where two viewpoints can be photographed in order to photograph images of two viewpoints as a plurality of viewpoints. Yes.
- the cameras 41 and 42 are arranged at different positions on a straight line on a horizontal plane with the optical axis directed in a direction perpendicular to the straight line. I will do it.
- the camera 41 is disposed on the right side of the camera 42 (toward the subject) (toward the photographing direction), and thus the camera 42 is disposed on the left side of the camera 41.
- the camera 41 shoots a subject at a position where the camera 41 is arranged, and outputs a color image C # 1 that is a moving image.
- the camera 41 outputs, for example, a disparity vector d1 representing the disparity with respect to the reference viewpoint for each pixel of the color image C # 1, using the position of the camera 42, which is another arbitrary camera, as the reference viewpoint. To do.
- the camera 42 shoots a subject at a position where the camera 42 is arranged, and outputs a color image C # 2 that is a moving image.
- the camera 42 outputs a disparity vector d2 representing the disparity with respect to the reference viewpoint for each pixel of the color image C # 2 with the position of the camera 41 as another arbitrary camera as the reference viewpoint. To do.
- the disparity vectors d1 and d2 are vectors whose values are corresponding to the positional relationship of the cameras 41 and 42 in the horizontal direction, etc., with the y component being 0.
- parallax vectors d1 and d2 output from the cameras 41 and 42 are also referred to as shooting parallax vectors d1 and d2.
- the color image C # 1 and the shooting parallax vector d1 output from the camera 41 and the color image C # 2 and the shooting parallax vector d2 output from the camera 42 are supplied to the multi-viewpoint image information generation unit 43. .
- the multi-viewpoint image information generation unit 43 outputs the color image C # 1 from the cameras 41 and 42 as it is.
- the multi-viewpoint image information generation unit 43 obtains parallax information regarding the parallax for each pixel of the color image # 1 from the shooting parallax vector d1 from the camera 41, and the parallax information image D # having the parallax information as a pixel value Generate 1 and output.
- the multi-viewpoint image information generation unit 43 obtains parallax information regarding the parallax for each pixel of the color image # 2 from the shooting parallax vector d2 from the camera 42, and the parallax information image D # having the parallax information as a pixel value Generate 2 and output.
- the disparity information for example, the disparity value (value I) corresponding to the shooting disparity vector and the depth Z representing the distance (depth) to the subject are normalized. Depth value (value y).
- the pixel value of the parallax information image takes an integer value of 0 to 255 represented by 8 bits, for example.
- the shooting parallax vector (x component thereof) (parallax) is represented by d and the maximum value of the shooting parallax vector (x component thereof) (parallax) (for example, in a picture or a moving image as one content)
- the minimum values are expressed as dmax (Dmax) and dmin (Dmin), respectively.
- the parallax value ⁇ (value I) is obtained using, for example, the shooting parallax vector (x component thereof) (parallax) d, and its maximum value dmax (Dmax) and minimum value dmin (Dmin). Is obtained according to the equation (3).
- parallax value ⁇ in Expression (3) can be converted into a shooting parallax vector (its x component) d according to Expression (4).
- the depth Z represents the distance from the straight line where the cameras 41 and 42 are arranged to the subject.
- the base line length which is the distance (distance from the reference viewpoint) between the camera 41 and the camera 42 arranged in a straight line, is L, and the focal length of the camera 41 is f.
- the depth Z can be obtained according to the equation (5) using the imaging parallax vector (x component thereof) d (d1).
- the disparity value ⁇ (value I) and the depth Z, which are disparity information (depth information), are equivalent information because they can be converted into each other using Expressions (3) to (5).
- a parallax information image having a parallax value ⁇ (value I) as a pixel value is also referred to as a parallax image, and an image having a depth value (value y) as a pixel value is referred to as a depth image.
- a parallax image of a parallax image and a depth image is used as the parallax information image (depth image), but a depth image is used as the parallax information image (depth image). It is also possible.
- the multi-viewpoint image information generation unit 43 outputs parallax related information in addition to the color images # 1 and # 2 and the parallax images D # 1 and # 2.
- the multi-viewpoint image information generation unit 43 includes a base line length L, a focal length f, and a camera distance from the outside (distances between the cameras 41 and 42 and the reference viewpoint) from the outside. Position information representing the positional relationship between 41 and 42 is supplied.
- the position information represents, for example, whether the cameras 41 and 42 are located on the left side or the right side with respect to the cameras 42 and 41 serving as the reference viewpoint.
- the position information indicates that the camera 41 is located on the right side of the camera 42 (the camera 42 is located on the left side of the camera 41).
- the viewpoint # 1 is located on the right side and the viewpoint # 2 is located on the left side.
- the multi-viewpoint image information generation unit 43 sets the maximum value dmax and the minimum value dmin of the shooting parallax vector (x component thereof) d for each of the shooting parallax vector d1 from the camera 41 and the shooting parallax vector d2 from the camera 41. To detect.
- the multi-viewpoint image information generation unit 43 outputs the maximum value dmax and the minimum value dmin of the shooting parallax vector d, the base line length L, the focal length f, and the position information as parallax related information.
- the color images C # 1 and C # 2, the parallax images D # 1 and D # 2, and the parallax related information output from the multiview image information generation unit 43 are supplied to the multiview image encoder in FIG.
- the cameras 41 and 42 are arranged on a straight line on the same plane orthogonal to the color image plane, and the shooting parallax vector d (d1 and d2) has a y component of 0.
- the cameras 41 and 42 can be arranged on different planes orthogonal to the color image plane.
- the shooting parallax vector d is a vector that can be a value other than 0 for both the x component and the y component.
- FIG. 7 is a diagram for explaining a picture to be referred to when a predicted image is generated in MVC predictive coding.
- the picture of the viewpoint # 1 image which is the base view image, is expressed as (display) time order as p11, p12, p13,...
- the viewpoint # 2 image which is the dependent view image, is displayed.
- the pictures are expressed as p21, p22, p23,.
- the base view picture for example, the picture p12 is predictively encoded by referring to the base view picture, for example, the pictures p11 and p13 as necessary.
- prediction generation of a predicted image
- prediction can be performed with reference to only the pictures p11 and p13 which are pictures at other times of the base view.
- the dependent view picture for example, the picture p22 requires the dependent view picture, for example, the pictures p21 and p23, and further the base view picture p12, which is another view, is required.
- the prediction encoding is performed with reference to.
- the dependent-view picture p22 performs prediction by referring to the pictures p21 and p23 that are pictures at other times of the dependent view and the base-view picture p12 that is a picture of another view. be able to.
- prediction performed with reference to a picture in the same view as the picture to be encoded is also referred to as temporal prediction
- prediction performed with reference to a picture in a view different from the picture to be encoded is referred to as disparity prediction.
- temporal prediction and disparity prediction can be performed for a dependent view picture.
- a picture of a view different from the encoding target picture that is referred to in the disparity prediction must be a picture at the same time as the encoding target picture.
- the encoders 11, 12, 21, and 22 constituting the multi-view image encoder in FIG. 5 perform prediction (prediction image generation) according to the MVC method.
- FIG. 8 is a diagram for explaining the encoding (and decoding) order of pictures in the MVC system.
- the picture of the viewpoint # 1 image that is the base view image is represented as p11, p12, p13,... In the (display) time order, and the viewpoint # that is the dependent view image.
- the pictures of the second image are represented as p21, p22, p23,.
- the base view picture and the dependent view picture are encoded in the same order.
- FIG. 9 is a block diagram illustrating a configuration example of the encoder 11 of FIG.
- encoders 12 and 21 in FIG. 5 are also configured in the same manner as the encoder 11 and encode an image in accordance with, for example, the MVC method.
- an encoder 11 includes an A / D (Analog / Digital) conversion unit 111, a screen rearrangement buffer 112, a calculation unit 113, an orthogonal transformation unit 114, a quantization unit 115, a variable length encoding unit 116, and a storage buffer 117. , An inverse quantization unit 118, an inverse orthogonal transform unit 119, a calculation unit 120, a deblocking filter 121, an intra prediction unit 122, an inter prediction unit 123, and a predicted image selection unit 124.
- a / D Analog / Digital
- the A / D conversion unit 111 is sequentially supplied with pictures of the color image C # 1 of the viewpoint # 1, which is an image to be encoded (moving image), in the display order.
- the A / D converter 111 When the picture supplied to the A / D converter 111 is an analog signal, the A / D converter 111 performs A / D conversion on the analog signal and supplies it to the screen rearrangement buffer 112.
- the screen rearrangement buffer 112 temporarily stores the pictures from the A / D conversion unit 111, and reads out the pictures according to a predetermined GOP (Group of Pictures) structure, thereby arranging the picture arrangement in the display order. From this, the rearrangement is performed in the order of encoding (decoding order).
- GOP Group of Pictures
- the picture read from the screen rearrangement buffer 112 is supplied to the calculation unit 113, the intra prediction unit 122, and the inter prediction unit 123.
- the calculation unit 113 is supplied with a picture from the screen rearrangement buffer 112 and a prediction image generated by the intra prediction unit 122 or the inter prediction unit 123 from the prediction image selection unit 124.
- the calculation unit 113 sets the picture read from the screen rearrangement buffer 112 as a target picture to be encoded, and sequentially sets macroblocks constituting the target picture as a target block to be encoded.
- the calculation unit 113 calculates a subtraction value obtained by subtracting the pixel value of the prediction image supplied from the prediction image selection unit 124 from the pixel value of the target block as necessary, and supplies the calculated value to the orthogonal transformation unit 114.
- the orthogonal transform unit 114 performs orthogonal transform such as discrete cosine transform and Karhunen-Loeve transform on the target block (the pixel value or the residual obtained by subtracting the predicted image) from the computation unit 113, and The transform coefficient obtained as a result is supplied to the quantization unit 115.
- the quantization unit 115 quantizes the transform coefficient supplied from the orthogonal transform unit 114, and supplies the quantized value obtained as a result to the variable length coding unit 116.
- variable length coding unit 116 performs variable length coding (for example, CAVLC (Context-Adaptive Variable Length Coding)) or arithmetic coding (for example, CABAC (Context) on the quantized value from the quantization unit 115. -Adaptive Binary Arithmetic Coding), etc.) and the like, and the encoded data obtained as a result is supplied to the accumulation buffer 117.
- variable length coding for example, CAVLC (Context-Adaptive Variable Length Coding)
- CABAC Context
- CABAC Context
- CABAC Context-Adaptive Binary Arithmetic Coding
- variable length coding unit 116 is also supplied with header information included in the header of the encoded data from the intra prediction unit 122 and the inter prediction unit 123. .
- variable length encoding unit 116 encodes the header information from the intra prediction unit 122 or the inter prediction unit 123 and includes it in the header of the encoded data.
- the accumulation buffer 117 temporarily stores the encoded data from the variable length encoding unit 116 and outputs it at a predetermined data rate.
- the encoded data output from the accumulation buffer 117 is supplied to the multiplexing unit 32 (FIG. 5).
- the quantization value obtained by the quantization unit 115 is supplied to the variable length coding unit 116 and also to the inverse quantization unit 118, and the inverse quantization unit 118, the inverse orthogonal transform unit 119, and the calculation In unit 120, local decoding is performed.
- the inverse quantization unit 118 inversely quantizes the quantized value from the quantization unit 115 into a transform coefficient and supplies the transform coefficient to the inverse orthogonal transform unit 119.
- the inverse orthogonal transform unit 119 performs inverse orthogonal transform on the transform coefficient from the inverse quantization unit 118 and supplies it to the arithmetic unit 120.
- the calculation unit 120 decodes the target block by adding the pixel value of the predicted image supplied from the predicted image selection unit 124 to the data supplied from the inverse orthogonal transform unit 119 as necessary. A decoded image is obtained and supplied to the deblocking filter 121.
- the deblocking filter 121 removes (reduces) block distortion generated in the decoded image by filtering the decoded image from the arithmetic unit 120, and supplies it to the DPB 31 (FIG. 5).
- the DPB 31 predictively encodes the decoded image from the deblocking filter 121, that is, the picture of the color image C # 1 encoded by the encoder 11 and locally decoded (timer 113).
- the reference picture is stored as a reference picture (candidate) to be referred to when generating a predicted picture to be used for encoding).
- the encoder 12 Since the DPB 31 is shared by the encoders 11, 12, 21, and 22, in addition to the picture of the color image C # 1 encoded and locally decoded by the encoder 11, the encoder 12 The picture of the color image C # 2 that has been encoded and locally decoded in FIG. 5, the picture of the disparity image D # 1 that has been encoded and locally decoded by the encoder 21, and the disparity that has been encoded by the encoder 22 and locally decoded The picture of image D # 2 is also stored.
- the DPB 31 stores decoded images of an I picture, a P picture, and a Bs picture.
- the in-screen prediction unit 122 reads from the DPB 31 the target picture. A portion (decoded image) that has already been locally decoded is read. Then, the intra-screen prediction unit 122 sets a part of the decoded image of the target picture read from the DPB 31 as the predicted image of the target block of the target picture supplied from the screen rearrangement buffer 112.
- the intra-screen prediction unit 122 calculates the encoding cost required to encode the target block using the predicted image, that is, the encoding cost required to encode the residual of the target block with respect to the predicted image. Obtained and supplied to the predicted image selection unit 124 together with the predicted image.
- the inter prediction unit 123 encodes from the DPB 31 one or more encoded and locally decoded before the target picture Are read as reference pictures.
- the inter prediction unit 123 uses the target block of the target picture from the screen rearrangement buffer 112 and the ME using the reference picture, and the corresponding block of the target block and the reference picture corresponding to the target block (for example, the target block) A deviation vector representing a deviation (parallax, motion) from a block (block that minimizes SAD (Sum of Absolute) Differences) is detected.
- the shift vector detected by the ME using the target block and the reference picture is between the target block and the reference picture.
- This is a motion vector representing a motion (temporal shift).
- the shift vector detected by the ME using the target block and the reference picture is a disparity (spatial) between the target block and the reference picture. This is a disparity vector representing a deviation.
- the parallax vector obtained by the ME is also referred to as a calculated parallax vector in order to distinguish it from the shooting parallax vector described in FIG.
- the shooting disparity vector is a vector whose y component is 0.
- the calculated disparity vector detected by the ME is the target block and the target picture of the reference picture. Since the shift (positional relationship) with a block (corresponding block) as an area that minimizes SAD or the like with the block is represented, the y component is not always zero.
- the inter prediction unit 123 performs prediction by performing reference picture deviation compensation from the DPB 31 (motion compensation that compensates for deviation of motion or parallax compensation that compensates for deviation of parallax) according to the deviation vector of the target block. Generate an image.
- the inter prediction unit 123 acquires, as a predicted image, a corresponding block that is a block (region) at a position shifted (shifted) from the position of the target block of the reference picture according to the shift vector of the target block.
- the inter prediction unit 123 sets the encoding cost required to encode the target block using a prediction image for each inter prediction mode in which a reference picture used for generating a prediction image, a macroblock type described later, and the like are different. Ask.
- the inter prediction unit 123 sets the inter prediction mode with the minimum encoding cost as the optimal inter prediction mode that is the optimal inter prediction mode, and the prediction image and the encoding cost obtained in the optimal inter prediction mode.
- the predicted image selection unit 124 is supplied.
- the predicted image selection unit 124 selects a predicted image having a lower encoding cost from the predicted images from the intra-screen prediction unit 122 and the inter prediction unit 123, and supplies the selected one to the calculation units 113 and 120.
- the intra prediction unit 122 supplies information related to intra prediction to the variable length encoding unit 116 as header information
- the inter prediction unit 123 includes information related to inter prediction (information on shift vectors and reference pictures). The assigned reference index or the like) is supplied to the variable length coding unit 116 as header information.
- variable length encoding unit 116 selects header information from the one in which the prediction image with the lower encoding cost is generated among the header information from the intra prediction unit 122 and the inter prediction unit 123, and Included in the header of the data.
- FIG. 10 is a diagram for explaining a macroblock type of the MVC (AVC) system.
- the macroblock is a block of horizontal ⁇ vertical 16 ⁇ 16 pixels.
- ME and generation of a predicted image
- the macroblock is divided into any partition of 16 ⁇ 16 pixels, 16 ⁇ 8 pixels, 8 ⁇ 16 pixels, or 8 ⁇ 8 pixels, and ME is performed for each partition.
- a grazing vector motion vector or calculated disparity vector
- an 8 ⁇ 8 pixel partition is further divided into any one of 8 ⁇ 8 pixel, 8 ⁇ 4 pixel, 4 ⁇ 8 pixel, or 4 ⁇ 4 pixel subpartitions.
- ME can be performed to detect a grazing vector (motion vector or calculated disparity vector).
- the macro block type represents what partition (and sub-partition) the macro block is divided into.
- the encoding cost of each macroblock type is calculated as the encoding cost of each inter prediction mode, and the inter prediction mode (macroblock type) with the minimum encoding cost is selected. Is selected as the optimal inter prediction mode.
- FIG. 11 is a diagram for explaining a prediction vector (PMV) of the MVC (AVC) method.
- a shift vector motion vector or calculated disparity vector
- a predicted image is generated using the shift vector.
- the shift vector Since the shift vector is necessary for decoding the image on the decoder side, it is necessary to encode the shift vector information and include it in the encoded data. However, if the shift vector is encoded as it is, the shift vector The amount of code increases and the coding efficiency may deteriorate.
- the macro block is divided into 8 ⁇ 8 pixel partitions, and each of the 8 ⁇ 8 pixel partitions is further divided into 4 ⁇ 4 pixel sub-partitions.
- a certain macro block X is a target block to be encoded.
- the target block X is divided into 16 ⁇ 16 pixel partitions (the target block X is used as a partition as it is).
- the prediction vector PMVX of the shift vector mvX of the target block X is the target of the macroblocks already encoded (in raster scan order) when the target block X is encoded.
- the shift vector mvA of the macroblock A adjacent on the block X the shift vector mvB of the macroblock B adjacent to the left, and the shift vector mvC of the macroblock C adjacent to the upper right, according to the equation (6) Calculated.
- med () represents the median (median) of the values in parentheses.
- the target block X is the macro block at the right end of the picture or the like, if the shift vector mvC of the macro block C is not available (unavailable), the target block X is replaced with the shift vector mvC.
- the prediction vector PMVX is calculated using the shift vector mvD of the macroblock D adjacent on the upper left.
- the calculation of the prediction vector PMVX according to the equation (6) is performed independently for each of the x component and the y component.
- the difference vector mvX of the target block X and the difference mvX ⁇ PMV between the prediction vector PMVX are included as header vector information in the header information.
- FIG. 12 is a diagram for further explaining the MVC (AVC) prediction vector.
- a method of generating a prediction vector in the MVC scheme differs depending on a reference index (hereinafter, also referred to as a prediction reference index) assigned to a reference picture used for generating a prediction image of a block (macroblock) around the target block. .
- a reference index hereinafter, also referred to as a prediction reference index assigned to a reference picture used for generating a prediction image of a block (macroblock) around the target block.
- a plurality of pictures when generating a predicted image, a plurality of pictures can be used as reference pictures.
- the reference picture is stored in a buffer called DPB after decoding (local decoding).
- DPB Downlink Prediction Picture
- short-term reference pictures used (for short-term reference)
- long-time reference pictures used for long-term reference
- Pictures that are not marked are marked as unreferenced pictures (unusedunfor reference), respectively.
- the DPB is managed by the FIFO (First In First Out) method, and the pictures stored in the DPB are released in order from the picture with the smallest frame_num (becomes non-reference pictures).
- FIFO First In First Out
- the I (Intra) picture, the P (Predictive) picture, and the Bs picture that is a reference B (Bi-directional ⁇ Predictive) picture are stored in the DPB as short-time reference pictures.
- the moving window memory management method does not affect the long-term reference picture stored in the DPB. That is, in the moving window memory management method, only the short-time reference pictures are managed by the FIFO method among the reference pictures.
- pictures stored in the DPB are managed using a command called MMCO (Memory management control operation).
- MMCO Memory management control operation
- a short-time reference picture is set as a non-reference picture, or a reference index for managing a long-time reference picture for a short-time reference picture.
- Assign a long-term frame index to set a short-term reference picture as a long-term reference picture, set a maximum long-term frame index, and set all reference pictures as non-reference pictures Etc. can be performed.
- inter prediction for generating a predicted image is performed by performing motion compensation of a reference picture stored in the DPB.
- inter prediction of a B picture (including a Bs picture) has a maximum of 2 pictures. Reference pictures can be used.
- the inter prediction using the two reference pictures is referred to as L0 (List) 0) prediction and L1 (List 1) prediction, respectively.
- L0 prediction, L1 prediction, or both L0 prediction and L1 prediction are used as inter prediction.
- L0 prediction is used as inter prediction.
- a reference picture that is referred to when generating a predicted image is managed by a reference list (Reference Picture List).
- a reference index that is an index for designating a reference picture to be referred to when generating a predicted image is assigned to the reference picture stored in the DPB.
- the reference index is assigned only for the L0 prediction.
- both the L0 prediction and the L1 prediction may be used as the inter prediction for the B picture. Is assigned to both the L0 prediction and the L1 prediction.
- the reference index for L0 prediction is also referred to as L0 index
- the reference index for L1 prediction is also referred to as L1 index.
- a reference index (L0 index) having a smaller value is assigned to the reference picture stored in the DPB as the reference picture is later in decoding order. It is done.
- the reference index is an integer value of 0 or more, and the minimum value is 0. Therefore, when the target picture is a P picture, 0 is assigned to the reference picture decoded immediately before the target picture as the L0 index.
- the reference index (L0 index, L0 index, POC (Picture Order Count) order is the reference picture stored in the DPB. And L1 index).
- an L0 index having a smaller value is assigned to a reference picture that is temporally previous to the target picture in the display order, and the reference picture that is closer to the target picture is allocated.
- an L0 index having a smaller value is assigned to a reference picture that is closer to the target picture.
- an L1 index with a smaller value is assigned to a reference picture that is temporally later than the target picture in display order, and the reference picture that is closer to the target picture is allocated.
- An L1 index having a smaller value is assigned to a reference picture that is closer to the current picture than a reference picture that is temporally previous.
- the default reference indexes (L0 index and L1 index) in the above AVC method are assigned to short-time reference pictures.
- the assignment of the reference index to the long-time reference picture is performed after the reference index is assigned to the short-time reference picture.
- a reference index having a larger value than that of the short-time reference picture is assigned to the long-time reference picture.
- the reference index can be assigned by using the default method as described above, or by using a command called Reference Picture List Reordering (hereinafter also referred to as RPLR command). it can.
- RPLR command Reference Picture List Reordering
- the reference index is assigned to the reference picture by a default method.
- the prediction vector PMVX of the shift vector mvX of the target block X is, as shown in FIG. 12, the macroblock A adjacent to the target block X, the macroblock B adjacent to the left, and the right Obtained in a different manner depending on the reference index for prediction of each macroblock C adjacent diagonally above (the reference index assigned to the reference picture used to generate each of the predicted images of macroblocks A, B, and C). It is done.
- the reference index ref_idx for prediction of the target block X is 0, for example.
- the shift vector of the one macroblock (the macroblock for which the prediction reference index ref_idx is 0) is set as the prediction vector PMVX of the shift vector mvX of the target block X.
- the macroblock A among the three macroblocks A to C adjacent to the target block X is a macroblock whose reference index ref_idx for prediction is 0.
- the shift vector mvA of the macroblock A is set as the prediction vector PMVX of the target block X (shift vector mvX).
- the median of the shift vector of two or more macroblocks for which the reference index ref_idx for prediction is 0 is set as the prediction vector PMVX of the target block X.
- all of the three macroblocks A to C adjacent to the target block X are macroblocks for which the reference index ref_idx for prediction is 0. Therefore, the shift vector of the macroblock A
- the median med (mvA, mvB, mvC) of the deviation vector mvB of the macro block B and the deviation vector mvC of the macro block C is set as the prediction vector PMVX of the target block X.
- the 0 vector is set as the prediction vector PMVX of the target block X.
- the target block when encoding a target block using a reference picture to which a reference index rev_idx having a value of 0 is assigned, the target block can be a skip macroblock.
- the prediction vector is directly used as the shift vector of the skip macroblock, and a copy of the block (corresponding block) of the reference picture at a position shifted by the shift vector from the position of the skip macroblock is the skip macroblock. The result of decoding the block.
- the target block is a skip macroblock depends on the specifications of the encoder, but is determined (determined) based on, for example, the amount of encoded data, the encoding cost of the target block, and the like.
- FIG. 13 is a block diagram illustrating a configuration example of the encoder 22 of FIG.
- the encoder 22 encodes the parallax image D # 2 of the viewpoint # 2 that is the image to be encoded using the MVC method, that is, as described in FIG.
- an encoder 22 includes an A / D conversion unit 211, a screen rearranging buffer 212, a calculation unit 213, an orthogonal transformation unit 214, a quantization unit 215, a variable length coding unit 216, an accumulation buffer 217, and an inverse quantization unit. 218, an inverse orthogonal transform unit 219, a calculation unit 220, a deblocking filter 221, an intra-screen prediction unit 222, a predicted image selection unit 224, a warping unit 231, a warped picture buffer 232, an occlusion detection unit 233, and a parallax prediction unit 234
- a / D conversion unit 211 includes an A / D conversion unit 211, a screen rearranging buffer 212, a calculation unit 213, an orthogonal transformation unit 214, a quantization unit 215, a variable length coding unit 216, an accumulation buffer 217, and an inverse quantization unit. 218, an inverse orthogonal transform unit 219, a calculation unit 220, a de
- the A / D conversion unit 211 to the intra prediction unit 222 and the prediction image selection unit 224 are the A / D conversion unit 111 to the intra prediction unit 122 and the prediction image selection unit 124 of the encoder 11 of FIG. Since each of them is configured similarly, the description thereof will be omitted as appropriate.
- the DPB 31 is supplied with a decoded image, that is, a picture of a parallax image (hereinafter also referred to as a decoded parallax image) D # 2 encoded by the encoder 22 and locally decoded from the deblocking filter 221. Stored as a reference picture.
- a decoded image that is, a picture of a parallax image (hereinafter also referred to as a decoded parallax image) D # 2 encoded by the encoder 22 and locally decoded from the deblocking filter 221.
- the DPB 31 includes a picture of the color image C # 1 encoded by the encoder 11 and locally decoded, and a color image C encoded by the encoder 12 and locally decoded.
- the # 2 picture and the parallax image (decoded parallax image) D # 1 encoded and locally decoded by the encoder 21 are also supplied and stored.
- the decoded parallax image D # 1 obtained by the encoder 21 is used for encoding the parallax image D # 2 to be encoded. Therefore, in FIG. 13, an arrow indicating that the decoded parallax image D # 1 obtained by the encoder 21 is supplied to the DPB 31 is illustrated.
- the warping unit 231 includes the maximum value dmax and the minimum value dmin of the shooting parallax vector d (the shooting parallax vector d1 of the viewpoint # 1), the baseline length L, the focal length f, and the position as parallax related information (FIG. 5). Information is supplied.
- the warping unit 231 acquires (reads out) a picture of the decoded parallax image D # 1 (a picture at the same time as the target picture) among the decoded parallax images D # 1 and D # 2 stored in the DPB 31. .
- the warping unit 231 warps the picture of the decoded parallax image D # 1 acquired from the DPB 31 using the parallax related information as necessary, and thereby the picture of the decoded parallax image D # 1 at the viewpoint # 2.
- the warping unit 231 uses the maximum value dmax and the minimum value dmin of the shooting parallax vector d as the pixel value of each pixel of the picture of the decoded parallax image D # 1, and uses the maximum value dmax and the minimum value dmin to calculate the pixel Each is converted into a shooting parallax vector d (d1).
- the base line length L and the focal length f are used, and the normality of the depth value, which is the pixel value of the depth image, according to Equation (5)
- the depth Z before conversion is converted into a shooting parallax vector d.
- the warping unit 231 generates a picture of the warped parallax image D ′ # 1 by performing warping that moves each pixel of the picture of the decoded parallax image D # 1 according to the shooting parallax vector d of the pixel.
- the warping unit 231 When the warping unit 231 generates a picture of the warped parallax image D ′ # 1 by warping the picture of the decoded parallax image D # 1, the warping unit 231 supplies the warped parallax image D ′ # 1 picture to the warped picture buffer 232. .
- the disparity related information supplied to the warping unit 231 is also supplied to the variable length encoding unit 216 and included in the header of the encoded data as header information as necessary.
- disparity related information can be multiplexed together with the encoded data in the multiplexing unit 32 (FIG. 5) instead of being included in the header of the encoded data.
- the warped picture buffer 232 temporarily stores the picture of the warped parallax image D ′ # 1 from the warping unit 231.
- a warped picture buffer 232 that stores a picture of the warped parallax image D ′ # 1 is provided separately from the DPB 31, but the DPB 31 and the warped picture buffer 232 are one buffer. It is possible to use both.
- the occlusion detection unit 233 has a portion (pixel) that has no pixel value in the picture of the warped parallax image D ′ # 1 stored in the warped picture buffer 232 (the picture at the same time as the target picture). Detect the part with no value.
- the occlusion detection unit 233 detects a portion of the target picture of the parallax image D # 2 at the same position as the portion without a pixel value as an occlusion portion.
- the occlusion detection unit 233 starts the occlusion portion by scanning each block that can be processed in the target picture of the parallax image D # 2 in the horizontal direction (with parallax). Detecting start pixel S and ending end pixel E are detected.
- the occlusion detection unit 233 When the occlusion detection unit 233 detects the start pixel S and the end pixel E of each block of the target picture, the occlusion detection unit 233 supplies the start pixel S and the end pixel E to the parallax prediction unit 234 as occlusion information.
- the parallax prediction unit 234 performs parallax prediction (generation of a predicted image) of the target block using the picture of the decoded parallax image D # 1 of the viewpoint # 1 stored in the DPB 31 as a reference picture.
- the disparity prediction unit 234 obtains a calculated disparity vector of the target block by performing ME using the picture of the decoded disparity image D # 1 stored in the DPB 31 as a reference picture.
- the disparity prediction unit 234 performs MC (Motion Compensation) (motion compensation) using the picture of the decoded disparity image D # 1 stored in the DPB 31 as a reference picture according to the calculated disparity vector of the target block, thereby A predicted image is generated.
- MC Motion Compensation
- motion compensation motion compensation
- the disparity prediction unit 234 calculates, for each macroblock type, an encoding cost required for encoding the target block (predictive encoding) using a prediction image obtained by disparity prediction from the reference picture.
- the disparity prediction unit 234 selects the macroblock type with the lowest coding cost as the optimal inter prediction mode, and supplies the prediction image generated in the optimal inter prediction mode to the prediction image selection unit 224.
- the disparity prediction unit 234 outputs information such as the optimal inter prediction mode to the variable length encoding unit 216 as header information.
- a reference index is assigned to the reference picture, and the reference picture is assigned to the reference picture that is referred to when the predicted image generated in the optimal inter prediction mode is generated in the disparity prediction unit 234.
- the reference index is selected as a reference index for prediction of the target block, and is output to the variable length coding unit 216 as one piece of header information.
- the disparity prediction unit 234 obtains a prediction vector (calculated disparity vector) of the target block using the occlusion information from the occlusion detection unit 463 as necessary. Then, the disparity prediction unit 234 obtains a residual vector that is a difference between the calculated disparity vector of the target block and the prediction vector, and outputs the residual vector to the variable length coding unit 216 as one piece of header information.
- the encoder 22 is provided with a parallax prediction unit 234 that performs only the parallax prediction of the inter prediction. Similar to the prediction unit 123, temporal prediction can be performed in addition to parallax prediction.
- a reference index is assigned to a picture of D # 2 (another time picture having a different time from the target picture).
- the encoder 22 is referred to, for example, to generate a prediction image having a smaller coding cost of the target block, among a prediction image generated by disparity prediction and a prediction image generated by temporal prediction.
- the reference index assigned to the reference picture is selected as the reference index for prediction of the target block, and is used as one piece of header information.
- FIG. 14 is a block diagram illustrating a configuration example of the disparity prediction unit 234 in FIG.
- the parallax prediction unit 234 includes a parallax detection unit 241, a parallax compensation unit 242, a prediction vector generation unit 243, a cost function calculation unit 244, and a mode selection unit 245.
- the parallax detection unit 241 is supplied with a picture of the decoded parallax image D # 1 which is a reference picture stored in the DPB 31, and also from the screen rearrangement buffer 212, a picture of the parallax image D # 2 to be encoded (target Picture).
- the disparity detection unit 241 performs ME using the picture of the decoded disparity image D # 1 as a reference picture for the target block in the target picture from the screen rearrangement buffer 212, as in the MVC scheme, In the picture of the decoded parallax image D # 1, for example, a shift vector representing a shift from the corresponding block that minimizes the SAD with the target block, that is, a calculated parallax vector mv representing the parallax with respect to the viewpoint # 1 of the target block, Detect for each macroblock type.
- the block that minimizes the SAD with the target block is the corresponding block used to obtain the calculated parallax vector mv.
- D represents the SAD of the target block and the block at the position moved by the calculated disparity vector mv from the target block of the picture of the decoded parallax image D # 1, and ⁇ is for R It is a weight and is set according to the quantization step of the residual of the target block (residual between the target block and the predicted image).
- R is a value corresponding to the code amount of the calculated disparity vector mv.
- the disparity detection unit 241 When adopting a value corresponding to the code amount of the residual vector as the value R, the disparity detection unit 241 recognizes the prediction vector from the prediction vector information supplied from the prediction vector generation unit 243, and the prediction vector, A residual vector is obtained from the calculated parallax vector mv.
- the disparity detection unit 241 detects the calculated disparity vector mv of the target block, the disparity detection unit 241 calculates a residual vector that is a difference between the calculated disparity vector mv and the prediction vector information obtained from the prediction vector information supplied from the prediction vector generation unit 243. It is obtained and supplied to the parallax compensation unit 242.
- the parallax compensation unit 242 is supplied with the residual vector of the calculated parallax vector mv from the parallax detection unit 241 and also the picture of the decoded parallax image D # 1 that is the reference picture stored in the DPB 31.
- the prediction vector information is supplied from the prediction vector generation unit 243 to the parallax compensation unit 242.
- the parallax compensation unit 242 restores the calculated parallax vector mv of the target block from the residual vector from the parallax detection unit 241 and the prediction vector information from the prediction vector generation unit 243.
- the disparity compensation unit 242 uses the picture of the decoded disparity image D # 1 from the DPB 31 as a reference picture, performs displacement compensation (disparity compensation) of the reference picture, and uses the calculated disparity vector mv of the target block. By performing in the same manner as above, a predicted image of the target block is generated for each macroblock type.
- the disparity compensation unit 242 acquires, as a predicted image, a corresponding block that is a block (region) at a position shifted by the calculated disparity vector mv from the position of the target block in the picture of the decoded disparity image D # 1.
- the parallax compensation unit 242 allocates the predicted image to the residual vector from the parallax detection unit 241 and the reference picture used to generate the predicted image (here, the picture of the decoded parallax image D # 1). It is supplied to the cost function calculation unit 244 together with the reference index.
- the prediction vector generation unit 243 is supplied with the start pixel S and the end pixel E as occlusion information from the occlusion detection unit 233.
- the predicted vector generation unit 243 determines whether the target block is an occlusion block using the occlusion information from the occlusion detection unit 233.
- generation part 243 produces
- the prediction vector generation unit 243 generates a prediction vector of the target block using the proximity prediction method described with reference to FIGS.
- the prediction vector generation unit 243 when the target block is not an occlusion block (when it is a non-occlusion block), the prediction vector generation unit 243 generates a prediction vector of the target block using the MVC method described with reference to FIGS.
- the prediction vector generation unit 243 generates a prediction vector for each macroblock type (FIG. 10), and supplies the prediction vector information to the parallax detection unit 241, the parallax compensation unit 242, and the cost function calculation unit 244.
- the cost function calculation unit 244 is supplied with a prediction image, a residual vector, and a reference index from the parallax compensation unit 242 and is also supplied with prediction vector information from the prediction vector generation unit 243 and screen rearrangement.
- the target picture of the parallax image D # 2 is supplied from the unit buffer 212.
- the cost function calculation unit 244 determines, for each macroblock type (FIG. 10), the encoding cost required for encoding the target block of the target picture from the screen rearrangement buffer 212 according to a predetermined cost function for calculating the encoding cost. Ask.
- the cost function calculation unit 244 obtains a value MV corresponding to the code amount of the residual vector from the parallax compensation unit 242, and corresponds to the code amount of the reference index (prediction reference index) from the parallax compensation unit 242. Find the value IN.
- the cost function calculation unit 244 obtains SAD that is a value D corresponding to the residual of the target block with respect to the predicted image from the parallax compensation unit 242.
- the cost function calculation unit 244 When the cost function calculation unit 244 obtains the coding cost (cost function value) for each macroblock type, the cost function calculation unit 244 supplies the coding cost to the mode selection unit 245 together with the reference index, the predicted image, and the residual vector.
- the mode selection unit 245 detects the minimum cost, which is the minimum value, from the encoding costs for each macroblock type from the cost function calculation unit 244.
- the mode selection unit 245 selects the macro block type for which the minimum cost is obtained as the optimal inter prediction mode.
- the mode selection unit 245 uses, as header information, mode-related information representing the optimal inter prediction mode, a reference index of the optimal inter prediction mode (reference index for prediction), a residual vector, and the like as variable length codes. To the conversion unit 216.
- the mode selection unit 245 supplies the prediction image and the encoding cost (minimum cost) in the optimal inter prediction mode to the prediction image selection unit 224.
- the mode selection unit 245 encodes the target block as a skip macroblock based on the minimum cost or the like, for example. Judge whether or not.
- the optimal inter prediction mode is set to a skip mode in which the target block is encoded as a skip macroblock.
- FIG. 15 is a flowchart for describing an encoding process for encoding the parallax image D # 2 of the viewpoint # 2 performed by the encoder 22 of FIG.
- step S11 the A / D conversion unit 211 performs A / D conversion on the analog signal of the picture of the parallax image D # 2 of viewpoint # 2 supplied thereto, and supplies the analog signal to the screen rearrangement buffer 212.
- the process proceeds to step S12.
- step S12 the screen rearrangement buffer 212 temporarily stores the picture of the parallax image D # 2 from the A / D conversion unit 211, and reads out the picture according to a predetermined GOP structure, thereby Rearrangement is performed to rearrange the display from the display order to the encoding order (decoding order).
- the picture read from the screen rearrangement buffer 212 is supplied to the calculation unit 213, the intra-screen prediction unit 222, and the parallax prediction unit 234, and the process proceeds from step S12 to step S13.
- step S13 the calculation unit 213 sets the picture of the parallax image D # 2 from the screen rearrangement buffer 212 as a target picture to be encoded, and further sequentially selects macroblocks constituting the target picture as encoding target pictures.
- the target block The target block.
- the calculation unit 213 calculates a difference (residual) between the pixel value of the target block and the pixel value of the predicted image supplied from the predicted image selection unit 224 as necessary, and supplies the difference to the orthogonal transform unit 214. Then, the process proceeds from step S13 to step S14.
- step S14 the orthogonal transform unit 214 performs orthogonal transform on the target block from the calculation unit 213, supplies the transform coefficient obtained as a result to the quantization unit 215, and the process proceeds to step S15.
- step S15 the quantization unit 215 quantizes the transform coefficient supplied from the orthogonal transform unit 214, and supplies the resulting quantized value to the inverse quantization unit 218 and the variable length coding unit 216. Then, the process proceeds to step S16.
- step S16 the inverse quantization unit 218 inversely quantizes the quantized value from the quantization unit 215 into a transform coefficient, supplies the transform coefficient to the inverse orthogonal transform unit 219, and the process proceeds to step S17.
- step S17 the inverse orthogonal transform unit 219 performs inverse orthogonal transform on the transform coefficient from the inverse quantization unit 218, supplies the transform coefficient to the calculation unit 220, and the process proceeds to step S18.
- step S18 the calculation unit 220 adds the pixel value of the predicted image supplied from the predicted image selection unit 224 to the data supplied from the inverse orthogonal transform unit 219 as necessary, thereby adding the target block.
- a decoded parallax image D # 2 obtained by decoding (local decoding) is obtained.
- the arithmetic unit 220 supplies the decoded parallax image D # 2 obtained by locally decoding the target block to the deblocking filter 221, and the processing proceeds from step S18 to step S19.
- step S19 the deblocking filter 221 filters the decoded parallax image D # 2 from the calculation unit 220 and supplies it to the DPB 31 (FIG. 5), and the process proceeds to step S20.
- step S20 the DPB 31 is supplied with the decoded parallax image D # 1 obtained by encoding the parallax image D # 1 and performing local decoding from the encoder 21 that encodes the parallax image D # 1. After waiting, the decoded parallax image D # 1 is stored, and the process proceeds to step S21.
- step S21 the DPB 31 stores the decoded parallax image D # 2 from the deblocking filter 221, and the process proceeds to step S22.
- step S 22 the warping unit 231 generates a picture of the warped parallax image D ′ # 1 by warping the picture of the decoded parallax image D # 1 stored in the DPB 31, and supplies the generated picture to the warped picture buffer 232.
- the process proceeds to step S23.
- step S23 the warped picture buffer 232 stores the picture of the warped parallax image D ′ # 1 from the warping unit 231, and the process proceeds to step S24.
- step S24 the occlusion detection unit 233 uses the picture of the warped parallax image D ′ # 1 stored in the warped picture buffer 232 to perform an occlusion detection process for detecting the occlusion portion of the target picture of the parallax image D # 2. Do.
- the occlusion detection unit 233 supplies the occlusion information obtained by the occlusion detection process to the parallax prediction unit 234, and the process proceeds from step S24 to step S25.
- step S25 the intra prediction unit 222 performs an intra prediction process (intra prediction process) for the next target block that is the next macro block to be encoded.
- the intra prediction unit 222 performs intra prediction (intra prediction) for generating a prediction image (prediction image of intra prediction) from the picture of the decoded parallax image D # 2 stored in the DPB 31 for the next target block. Do.
- the intra-screen prediction unit 222 obtains the encoding cost required to encode the target block using the prediction image of the intra prediction, and supplies it to the prediction image selection unit 224 together with the prediction image of the intra prediction. The process proceeds from step S25 to step S26.
- step S26 the disparity prediction unit 234 performs disparity prediction processing on the next target block using the picture of the decoded disparity image D # 1 stored in the DPB 31 as a reference picture.
- the disparity prediction unit 234 performs disparity prediction on the next target block using the decoded disparity image D # 1 stored in the DPB 31 as a reference image for each inter prediction mode with different macroblock types and the like. Then, a calculated parallax vector, a prediction vector, a prediction image, an encoding cost, and the like are obtained.
- the disparity prediction unit 234 sets the inter prediction mode with the minimum encoding cost as the optimal inter prediction mode, and supplies the prediction image of the optimal inter prediction mode together with the encoding cost to the prediction image selection unit 224. The process proceeds from step S26 to step S27.
- the predicted image selection unit 224 includes, for example, a predicted image from the intra-screen prediction unit 222 (prediction image for intra prediction) and a predicted image from the parallax prediction unit 234 (prediction image for inter prediction).
- the prediction image with the smaller encoding cost is selected and supplied to the calculation units 213 and 220, and the process proceeds to step S28.
- the predicted image selected by the predicted image selection unit 224 in step S27 is used in the processing of steps S13 and S18 performed in the encoding of the next target block.
- the intra-screen prediction unit 222 supplies information related to the intra prediction obtained in the intra prediction process of step S25 as header information to the variable length encoding unit 216, and the disparity prediction unit 234 performs the disparity prediction process of step S26.
- Information related to the parallax prediction (inter prediction) obtained in is supplied to the variable length coding unit 216 as header information.
- step S28 the variable length encoding unit 216 performs variable length encoding on the quantized value from the quantization unit 215 to obtain encoded data.
- variable length coding unit 216 selects header information from the one in which the prediction image with the lower coding cost is generated among the header information from each of the intra-screen prediction unit 222 and the parallax prediction unit 234. Included in the header of the encoded data.
- variable length encoding unit 216 supplies the encoded data to the accumulation buffer 217, and the process proceeds from step S28 to step S29.
- step S29 the accumulation buffer 217 temporarily stores the encoded data from the variable length encoding unit 216 and outputs it at a predetermined data rate.
- the encoded data output from the accumulation buffer 217 is supplied to the multiplexing unit 32 (FIG. 5).
- FIG. 16 is a flowchart illustrating the occlusion detection process performed by the occlusion detection unit 233 of FIG. 13 in step S24 of FIG.
- step S31 the occlusion detection unit 233 determines a portion without a pixel value (a portion without a pixel value) in the picture of the warped parallax image D ′ # 1 stored in the warped picture buffer 232. To detect.
- the occlusion detection unit 233 detects a portion of the parallax image D # 2 that is at the same position as the portion without a pixel value as an occlusion portion, and the process proceeds from step S31 to step S32.
- step S32 the occlusion detection unit 233 scans the target picture of the parallax image D # 2, for example, for each horizontal line constituting the macroblock, and the occlusion part (horizontal direction (x direction)) in the target picture.
- the occlusion start position SS and the occlusion end position EE which are the start position and the end position, are detected.
- the occlusion detection unit 233 scans the first horizontal line from the top of the target picture in the x direction (left to right), and the macroblock sequence (slice) in the first row from the top of the target picture. , The pixel position where the occlusion part starts is detected as the occlusion start position SS, and the pixel position where the occlusion part ends is detected as the occlusion end position EE.
- the pixel position where the occlusion part starts is detected as the occlusion start position SS
- the pixel position where the occlusion part ends is detected as the occlusion end position EE.
- the occlusion detection unit 233 scans, for example, the 16 ⁇ (k ⁇ 1) +1) th horizontal line from the top of the target picture in the x direction, and the k-th macroblock from the top of the target picture.
- the pixel position where the occlusion part starts is detected as the occlusion start position SS
- the pixel position where the occlusion part ends is detected as the occlusion end position EE.
- the occlusion start position SS and the occlusion end position EE represent the number of pixels from the left (left end) of the target picture (the number of pixels from the left).
- step S32 when the occlusion start position SS and the occlusion end position EE are detected up to the bottom macroblock sequence of the target picture, the process proceeds from step S32 to step S33, and the occlusion detection unit 233 From the target picture, for example, a macroblock that has not yet been selected as the target block in the coding order is selected as the target block, and the process proceeds to step S34.
- step S34 the occlusion detection unit 233 determines that the block of interest is an occlusion block based on the occlusion start position SS and the occlusion end position EE detected for the sequence of macroblocks having the number of stages in the target picture. The process proceeds to step S35.
- step S35 the occlusion detection unit 233 determines whether the block of interest is an occlusion block.
- step S35 If it is determined in step S35 that the block of interest is an occlusion block, the process proceeds to step S36, and the occlusion detector 233 determines that the occlusion portion described in FIGS.
- the starting start pixel S and the finishing end pixel E are detected and stored, and the process proceeds to step S38.
- step S35 If it is determined in step S35 that the block of interest is not an occlusion block, the process proceeds to step S37, and the occlusion detection unit 233 sets the start pixel S and end pixel E (of the block of interest that is a non-occlusion block as The variable is set to a value indicating that the block is not an occlusion block (for example, a value such as 0 or ⁇ 1 that cannot be taken as the start pixel S and the end pixel E of the occlusion block), and the process proceeds to step S38.
- a value indicating that the block is not an occlusion block for example, a value such as 0 or ⁇ 1 that cannot be taken as the start pixel S and the end pixel E of the occlusion block
- step S38 the occlusion detection unit 233 determines whether the start pixel S and the end pixel E have been obtained using all macroblocks of the target picture as the target block.
- step S38 If it is determined in step S38 that all macroblocks of the target picture are not yet the target block, the process returns to step S33, and the target block is still selected as the target block in the coding order from the target picture. The macro block that has not been selected is newly selected as the target block, and the same processing is repeated thereafter.
- step S38 If it is determined in step S38 that all the macroblocks of the target picture are the target blocks, the process returns.
- the occlusion detection process is performed with the macroblock as the minimum unit, so that for each macroblock, the start pixel S at which the occlusion portion starts and the end pixel E at the end of the macroblock are detected.
- the occlusion detection process can also be performed in units of detecting the calculated parallax vector (units for performing ME) in the parallax prediction unit 234.
- the occlusion detection process can be performed in units of partitions (subpartitions) described with reference to FIG.
- the minimum unit for performing the occlusion detection process is a 4 ⁇ 4 pixel partition (subpartition).
- FIG. 17 is a diagram illustrating the occlusion detection process (FIG. 16) performed by the occlusion detection unit 233 (FIG. 13).
- a in FIG. 17 indicates that the occlusion detection unit 233 receives the attention based on the occlusion start position SS and the occlusion end position EE detected in step S34 in FIG. It is a figure explaining the process which recognizes whether a block is an occlusion block.
- the viewpoint # 1 is located on the right side and the viewpoint # 2 is located on the left side.
- the background on the left side of the foreground is touched.
- the part is an occlusion part. Therefore, the left side of the occlusion part is the background, and the right side is the foreground.
- the position of the leftmost pixel of the macroblock is represented by SB, and the position of the rightmost pixel is represented by EB.
- the positions SB and EB represent the number of pixels from the left (left end) of the target picture (the number of pixels from the left). Therefore, EB-SB + 1 is 16 pixels, which is the width of the macroblock in the x direction.
- the occlusion start position SS and the occlusion end position EE represent the number of pixels from the left (left end) of the target picture (the number of pixels from the left).
- the position SB of the leftmost pixel of the macroblock and the position EB of the rightmost pixel are expressed by the formula SB ⁇ SS ⁇ EB ⁇ EE, the formula SS ⁇ SB ⁇ EB ⁇ EE, the formula SS ⁇ SB ⁇ EE ⁇ EB, and If any one of the formulas SB ⁇ SS ⁇ SE ⁇ EB is satisfied, the macroblock includes an occlusion portion.
- the occlusion detection unit 233 determines that the position SB of the leftmost pixel and the position EB of the rightmost pixel of the macroblock have the formula SB ⁇ SS ⁇ EB ⁇ EE, the formula SS ⁇ SB ⁇ EB ⁇ EE, and the formula SS ⁇ SB ⁇ EE. If any one of ⁇ EB and the formula SB ⁇ SS ⁇ SE ⁇ EB is satisfied, the macroblock is recognized as an occlusion block.
- blocks (macro blocks) B # 1, B # 2, and B # 3 are recognized as occlusion blocks.
- the occlusion detection unit 233 detects the start pixel S at which the occlusion part starts and the end pixel E at the end of the macro block that is an occlusion block in step S36 in FIG. FIG.
- the start pixel S # i at which the occlusion portion starts and the end pixel E # i at which it ends are pixels from the left of the macroblock. This represents the number (the number of the pixel from the left).
- the occlusion blocks B # 1, B # 2, and B # 3 are the second, third, and fourth blocks from the left of the target block, respectively.
- the occlusion start position SS of the occlusion portion is the 24th pixel and the occlusion end position EE is the 60th pixel.
- FIG. 18 is a flowchart illustrating the disparity prediction process performed by the disparity prediction unit 234 in FIG. 14 in step S26 in FIG.
- step S41 the prediction vector generation unit 243 generates the prediction vector of the (next) target block for each macroblock type (FIG. 10) using the occlusion information supplied from the occlusion detection unit 233 as necessary. A prediction vector generation process is performed.
- the prediction vector generation unit 243 supplies the prediction vector of the target block obtained by the prediction vector generation processing to the parallax detection unit 241, the parallax compensation unit 242, and the cost function calculation unit 244 as prediction vector information.
- the process proceeds from step S41 to step S42.
- step S42 the parallax prediction unit 234 acquires a picture of the decoded parallax image D # 1 that is a reference picture from the DPB 31, and supplies the decoded parallax image D # 1 to the parallax detection unit 241 and the parallax compensation unit 242. Proceed to
- step S43 the parallax detection unit 241 performs ME for the (next) target block in the target picture supplied from the screen rearrangement buffer 212, using the decoded parallax image D # 1 picture from the DPB 31 as a reference picture.
- the calculated disparity vector mv representing the disparity of the target block with respect to the viewpoint # 1 is detected.
- the disparity detection unit 241 obtains a residual vector that is a difference between the calculated disparity vector mv of the target block and the prediction vector obtained from the prediction vector information supplied from the prediction vector generation unit 243, and the disparity compensation unit 242 The process proceeds from step S43 to step S43.
- step S43 the parallax compensation unit 242 restores the calculated parallax vector mv of the target block from the residual vector from the parallax detection unit 241 and the prediction vector information from the prediction vector generation unit 243.
- the parallax compensation unit 242 generates a prediction image of the (next) target block by performing MC according to the calculated parallax vector mv of the target block using the decoded parallax image D # 1 picture from the DPB 31 as a reference picture. To do.
- the parallax compensation unit 242 acquires, as a predicted image, a corresponding block that is a block at a position shifted by the calculated parallax vector mv from the position of the target block in the picture of the decoded parallax image D # 1.
- the parallax compensation unit 242 allocates the predicted image to the residual vector from the parallax detection unit 241 and the reference picture used to generate the predicted image (here, the picture of the decoded parallax image D # 1). This is supplied to the cost function calculation unit 244 together with the reference index being processed, and the process proceeds from step S44 to step S45.
- step S45 the cost function calculation unit 244 determines, for each macroblock type (FIG. 10), the encoding cost required for encoding the target block of the target picture from the screen rearrangement buffer 212 according to a predetermined cost function.
- the reference index, the predicted image, and the residual vector from the parallax compensation unit 242 are supplied to the mode selection unit 245 together with the processing, and the process proceeds to step S46.
- step S46 the mode selection unit 245 detects the minimum cost, which is the minimum value, from the encoding costs for each macroblock type from the cost function calculation unit 244.
- the mode selection unit 245 selects the macro block type for which the minimum cost is obtained as the optimum inter prediction mode, and the process proceeds from step S46 to step S47.
- step S47 the mode selection unit 245 supplies the prediction image in the optimal inter prediction mode and the encoding cost (minimum cost) to the prediction image selection unit 224, and the process proceeds to step S48.
- step S48 the mode selection unit 245 uses, as header information, mode-related information indicating the optimal inter prediction mode, a reference index of the optimal inter prediction mode (reference index for prediction), and a residual vector of the optimal inter prediction mode. , And the process returns.
- FIG. 19 is a flowchart for explaining a prediction vector generation process performed by the prediction vector generation unit 243 (FIG. 14) in step S41 of FIG.
- the viewpoint # 1 is located on the right side and the viewpoint # 2 is located on the left side, and the parallax image D # 2 of the viewpoint # 2
- the figure shows a prediction vector generation process when the background part on the left side is an occlusion part, that is, when the left side of the occlusion part is the background and the right side is the foreground.
- the prediction vector generation unit 243 positions the viewpoint # 2 of the parallax image D # 2 to be encoded by the encoder 22 (FIG. 13) on the left side of the viewpoint # 1 of the parallax image D # 1 serving as a reference picture Is recognized from the position information supplied to the warping unit 231.
- the prediction vector generation unit 243 performs prediction vector generation processing according to the flowchart of FIG. 19 when the viewpoint # 2 is located on the left side of the viewpoint # 1.
- step S61 the prediction vector generation unit 243 acquires the start pixel S and the end pixel E of each block of the target picture as the occlusion information from the occlusion detection unit 233, and the process proceeds to step S62.
- step S62 the predicted vector generation unit 243 determines whether the target block is an occlusion block based on the occlusion information from the occlusion detection unit 233.
- the start pixel S and the end pixel E (variable as) of the non-occlusion block have values indicating that they are not occlusion blocks (start pixel S of the occlusion block). And a value such as 0 or -1 that cannot be taken as the end pixel E).
- the prediction vector generation unit 243 can determine whether the target block is an occlusion block based on the start pixel S and the end pixel E of the target block as the occlusion information.
- step S62 If it is determined in step S62 that the target block is not an occlusion block, that is, if the target block is a non-occlusion block, the process proceeds to step S63, and the prediction vector generation unit 243 performs the target in the MVC method. A block prediction vector PMV is generated, and the process returns.
- step S62 If it is determined in step S62 that the target block is an occlusion block, the process proceeds to step S64.
- the prediction vector PMV of the target block that is an occlusion block is illustrated in FIG. 2 and the proximity prediction method described in FIG.
- step S64 the prediction vector generation unit 243 determines whether the block adjacent to the left of the target block that is an occlusion block is a non-occlusion block based on the occlusion information.
- step S64 If it is determined in step S64 that the block on the left of the target block is not a non-occlusion block, that is, if the block on the left of the target block is an occlusion block, the process proceeds to step S66.
- FIG. 20 is a flowchart for explaining the prediction vector generation process performed by the prediction vector generation unit 243 (FIG. 14) in step S41 of FIG.
- the viewpoint # 1 is located on the left side
- the viewpoint # 2 is located on the right side
- the right side of the foreground is touched in the parallax image D # 2 of the viewpoint # 2.
- a prediction vector generation process is illustrated when the background portion is an occlusion portion, that is, when the right side of the occlusion portion is the background and the left side is the foreground.
- the prediction vector generation unit 243 positions the viewpoint # 2 of the parallax image D # 2 to be encoded by the encoder 22 (FIG. 13) on the right side of the viewpoint # 1 of the parallax image D # 1 serving as a reference picture Is recognized from the position information supplied to the warping unit 231.
- the prediction vector generation unit 243 performs prediction vector generation processing according to the flowchart of FIG. 20 when the viewpoint # 2 is located on the right side of the viewpoint # 1.
- step S71 the prediction vector generation unit 243 acquires the start pixel S and the end pixel E of each block of the target picture as the occlusion information from the occlusion detection unit 233, and the process proceeds to step S72.
- step S72 the prediction vector generation unit 243 determines whether the target block is an occlusion block based on the occlusion information from the occlusion detection unit 233.
- step S72 When it is determined in step S72 that the target block is not an occlusion block, that is, when the target block is a non-occlusion block, the process proceeds to step S73, and the prediction vector generation unit 243 performs the target in the MVC method. A block prediction vector PMV is generated, and the process returns.
- step S72 If it is determined in step S72 that the target block is an occlusion block, the process proceeds to step S74.
- the prediction vector PMV of the target block that is an occlusion block is It is generated by the proximity prediction method described in 4.
- step S74 the prediction vector generation unit 243 determines whether or not the block on the left of the target block that is an occlusion block is a non-occlusion block.
- step S74 If it is determined in step S74 that the block adjacent to the left of the target block is a non-occlusion block, that is, when the stage (slice) of the target block of the target picture is scanned in the x direction, If the occlusion part has started, the process proceeds to step S75, and the prediction vector generation unit 243 recognizes an occlusion block continuously arranged in the x direction starting from the target block from the occlusion information. The process proceeds to step S76.
- the i-th block from the top of the occlusion blocks arranged continuously in the x direction starting from the target block is represented as B # i. Further, it is assumed that the number of occlusion blocks that are continuously arranged in the x direction starting from the target block is N.
- the first block (target block) of the occlusion block that is continuously arranged in the x direction starting from the target block is represented by B # 1
- the last block is represented by B # N.
- step S76 the prediction vector generation unit 243 uses, as the occlusion information, start pixels S # 1 to S # 1 to B # 1 to B # N of all the occlusion blocks sequentially arranged in the x direction starting from the target block.
- start pixels S # 1 to S # 1 to B # 1 to B # N of all the occlusion blocks sequentially arranged in the x direction starting from the target block.
- step S74 If it is determined in step S74 that the block on the left of the target block is not a non-occlusion block, that is, if the block on the left of the target block is an occlusion block, the process proceeds to step S76.
- FIG. 21 is a block diagram illustrating a configuration example of an embodiment of a multi-view image decoder to which the present technology is applied.
- the multi-viewpoint image decoder in FIG. 21 is a decoder that decodes data obtained by encoding images of a plurality of viewpoints by using, for example, the MVC method. In the following, processing similar to the MVC method will be described as appropriate. Is omitted.
- multi-viewpoint image decoder is not limited to a decoder that uses the MVC method.
- the multiplexed data output from the multi-view image encoder of FIG. 5 is the color image C # 1 of the view # 1, which is the color image of the two views # 1 and # 2, and the view Decoded into color image C # 2 of # 2, parallax image D # 1 of viewpoint # 1, which is the parallax images of the two viewpoints # 1 and # 2, and parallax image D # 2 of viewpoint # 2 .
- the multi-viewpoint image decoder includes a separation unit 301, decoders 311, 312, 321, 322, and DPB 331.
- the multiplexed data output from the multi-viewpoint image encoder in FIG. 5 is supplied to the separation unit 301 via a recording medium and a transmission medium (not shown).
- the separation unit 301 from the multiplexed data supplied thereto, encodes color image C # 1, encoded data of color image C # 2, encoded data of parallax image D # 1, and parallax image D Separate encoded data of # 2.
- the separation unit 301 transmits the encoded data of the color image C # 1 to the decoder 311, the encoded data of the color image C # 2 to the decoder 312, and the encoded data of the parallax image D # 1 to the decoder 321.
- the encoded data of the image D # 2 is supplied to the decoder 322, respectively.
- the decoder 311 decodes the encoded data of the color image C # 1 from the separation unit 301, and outputs the color image C # 1 obtained as a result.
- the decoder 312 decodes the encoded data of the color image C # 2 from the separation unit 301, and outputs the color image C # 2 obtained as a result.
- the decoder 321 decodes the encoded data of the parallax image D # 1 from the separation unit 301 and outputs the parallax image D # 1 obtained as a result.
- the decoder 322 decodes the encoded data of the parallax image D # 2 from the separation unit 301, and outputs the parallax image D # 2 obtained as a result.
- the DPB 331 temporarily uses the decoded images (decoded images) obtained by decoding the decoding target images in the decoders 311, 312, 321, and 322 as reference picture candidates to be referred to when the predicted image is generated.
- the decoders 311, 312, 321, and 322 decode the images that have been predictively encoded by the encoders 11, 12, 21, and 22 in FIG. 5, respectively.
- the decoders 311, 312, 321, and 322 determine the predictive image used in the predictive encoding.
- the decoded image used for generating the predicted image is temporarily stored in the DPB 331.
- the DPB 331 is a shared buffer that temporarily stores decoded images (decoded images) obtained by the decoders 311, 312, 321, and 322.
- the decoders 311, 312, 321, and 322 are respectively connected to the DPB 331.
- a reference picture to be referenced for decoding the decoding target picture is selected from the stored decoded picture, and a predicted picture is generated using the reference picture.
- DPB 331 is shared by decoders 311, 312, 321 and 322, each of decoders 311, 312, 321 and 322 has a decoded image obtained by itself and a decoded image obtained by another decoder. Images can also be referenced.
- FIG. 22 is a block diagram illustrating a configuration example of the decoder 311 in FIG.
- decoders 312 and 321 in FIG. 21 are also configured in the same manner as the decoder 311 and encode an image in accordance with, for example, the MVC method.
- a decoder 311 includes an accumulation buffer 341, a variable length decoding unit 342, an inverse quantization unit 343, an inverse orthogonal transform unit 344, an operation unit 345, a deblocking filter 346, a screen rearrangement buffer 347, and a D / A conversion unit. 348, the intra prediction unit 349, the inter prediction unit 350, and the predicted image selection unit 351.
- the storage buffer 341 is supplied with encoded data of the color image C # 1 from the separation unit 301 (FIG. 21).
- the accumulation buffer 341 temporarily stores the encoded data supplied thereto and supplies the encoded data to the variable length decoding unit 342.
- variable length decoding unit 342 restores the quantization value and header information by variable length decoding the encoded data from the accumulation buffer 341. Then, the variable length decoding unit 342 supplies the quantization value to the inverse quantization unit 343 and supplies the header information to the intra-screen prediction unit 349 and the inter prediction unit 350.
- the inverse quantization unit 343 inversely quantizes the quantized value from the variable length decoding unit 342 into a transform coefficient and supplies the transform coefficient to the inverse orthogonal transform unit 344.
- the inverse orthogonal transform unit 344 performs inverse orthogonal transform on the transform coefficient from the inverse quantization unit 343, and supplies the transform coefficient to the arithmetic unit 345 in units of macroblocks.
- the calculation unit 345 sets the macroblock supplied from the inverse orthogonal transform unit 344 as a target block to be decoded, and adds the predicted image supplied from the predicted image selection unit 351 to the target block as necessary. Thus, a decoded image is obtained and supplied to the deblocking filter 346.
- the deblocking filter 346 performs, for example, the same filtering as the deblocking filter 121 in FIG. 9 on the decoded image from the arithmetic unit 345, and supplies the filtered decoded image to the screen rearrangement buffer 347.
- the screen rearrangement buffer 347 temporarily stores and reads out the picture of the decoded image from the deblocking filter 346, thereby rearranging the picture arrangement to the original arrangement (display order), and D / A (Digital / Analog) This is supplied to the conversion unit 348.
- the D / A conversion unit 348 performs D / A conversion on the picture when it is necessary to output the picture from the screen rearrangement buffer 347 as an analog signal and outputs the picture.
- the deblocking filter 346 supplies, to the DPB 331, decoded images of I pictures, P pictures, and Bs pictures, which are referenceable pictures, among the decoded images after filtering.
- the DPB 331 refers to a reference picture (referred to when a picture of the decoded image from the deblocking filter 346, that is, a picture of the color image C # 1 is generated in a prediction image used for decoding performed later in time. As a candidate).
- the DPB 331 is shared by the decoders 311, 312, 321, and 322, so that the color decoded by the decoder 312 in addition to the picture of the color image C # 1 decoded by the decoder 311.
- the picture of the image C # 2, the picture of the parallax image D # 1 decoded by the decoder 321 and the picture of the parallax image D # 2 decoded by the decoder 322 are also stored.
- the intra prediction unit 349 recognizes whether the target block is encoded using a prediction image generated by intra prediction (intra prediction) based on the header information from the variable length decoding unit 342.
- the intra-screen prediction unit 349 receives a picture including the target block from the DPB 331 as in the intra-screen prediction unit 122 of FIG. A portion (decoded image) that has already been decoded in the target picture) is read out. Then, the intra-screen prediction unit 349 supplies a part of the decoded image of the target picture read from the DPB 331 to the predicted image selection unit 351 as a predicted image of the target block.
- the inter prediction unit 350 recognizes whether the target block is encoded using a prediction image generated by the inter prediction based on the header information from the variable length decoding unit 342.
- the inter prediction unit 350 When the target block is encoded using a prediction image generated by inter prediction, the inter prediction unit 350, based on the header information from the variable length decoding unit 342, a prediction reference index, that is, the target block The reference index assigned to the reference picture used to generate the predicted image is recognized.
- the inter prediction unit 350 reads, as a reference picture, a reference picture to which a reference index for prediction is assigned from the reference pictures stored in the DPB 331.
- the inter prediction unit 350 recognizes a shift vector (disparity vector, motion vector) used for generating the predicted image of the target block based on the header information from the variable length decoding unit 342, and the inter prediction unit in FIG. Similarly to 123, a prediction picture is generated by performing reference picture shift compensation (motion compensation that compensates for a shift for motion or parallax compensation that compensates for a shift for parallax) according to the shift vector.
- a shift vector displacement vector, motion vector
- motion vector motion vector used for generating the predicted image of the target block based on the header information from the variable length decoding unit 342, and the inter prediction unit in FIG.
- a prediction picture is generated by performing reference picture shift compensation (motion compensation that compensates for a shift for motion or parallax compensation that compensates for a shift for parallax) according to the shift vector.
- the inter prediction unit 350 acquires, as a predicted image, a block (corresponding block) at a position moved (shifted) from the position of the target block of the reference picture according to the shift vector of the target block.
- the inter prediction unit 350 supplies the predicted image to the predicted image selection unit 351.
- the prediction image selection unit 351 selects the prediction image when the prediction image is supplied from the intra-screen prediction unit 349, and selects the prediction image when the prediction image is supplied from the inter prediction unit 350. And supplied to the calculation unit 345.
- FIG. 23 is a block diagram showing a configuration example of the decoder 322 in FIG.
- the decoder 322 performs decoding of the encoded data of the parallax image D # 2 of the viewpoint # 2 to be decoded by using the MVC method, that is, in the same manner as the local decoding performed by the encoder 22 of FIG.
- the decoder 322 includes an accumulation buffer 441, a variable length decoding unit 442, an inverse quantization unit 443, an inverse orthogonal transform unit 444, a calculation unit 445, a deblocking filter 446, a screen rearrangement buffer 447, and a D / A conversion unit. 448, an intra-screen prediction unit 449, a predicted image selection unit 451, a warping unit 461, a warped picture buffer 462, an occlusion detection unit 463, and a parallax prediction unit 464.
- the storage buffer 441 through the intra-screen prediction unit 449 and the predicted image selection unit 451 are configured in the same manner as the storage buffer 341 through the intra-screen prediction unit 349 and the predicted image selection unit 351 of FIG. The description is omitted as appropriate.
- the DPB 331 is supplied with a decoded image, that is, a picture of a decoded parallax image D # 2, which is a parallax image decoded by the decoder 322, from the deblocking filter 446, and is stored as a reference picture that can be a reference picture.
- a decoded image that is, a picture of a decoded parallax image D # 2, which is a parallax image decoded by the decoder 322, from the deblocking filter 446, and is stored as a reference picture that can be a reference picture.
- the DPB 331 includes a picture of the color image C # 1 decoded by the decoder 311, a picture of the color image C # 2 decoded by the decoder 312, and a decoder 321.
- the decoded parallax image (decoded parallax image) D # 1 picture is also supplied and stored.
- the decoded parallax image D # 1 obtained by the decoder 321 is used for decoding the parallax image D # 2 to be decoded. Therefore, in FIG. 23, an arrow indicating that the decoded parallax image D # 1 obtained by the decoder 321 is supplied to the DPB 331 is illustrated.
- the warping unit 461 receives the maximum value dmax and the minimum value dmin of the shooting parallax vector d (the shooting parallax vector d1 of the viewpoint # 1), the base line length L, the focal length f, In addition, position information is supplied.
- the warping unit 461 acquires (reads out) the picture of the decoded parallax image D # 1 among the pictures of the decoded parallax images D # 1 and D # 2 stored in the DPB 331.
- the warping unit 461 warps the picture of the decoded parallax image D # 1 acquired from the DPB 331 using the parallax related information as necessary, similarly to the warping unit 231 of FIG.
- a warped parallax image D ′ # 1 picture which is a warped image obtained by converting the D # 1 picture into an image (parallax image) obtained at the viewpoint # 2, is generated.
- the warping unit 461 When the warping unit 461 generates a picture of the warped parallax image D ′ # 1 by warping the picture of the decoded parallax image D # 1, the warping unit 461 supplies the picture of the warped parallax image D ′ # 1 to the warped picture buffer 462. .
- the warped picture buffer 462 temporarily stores the picture of the warped parallax image D ′ # 1 from the warping unit 461.
- a warped picture buffer 462 for storing a picture of the warped parallax image D ′ # 1 is provided separately from the DPB 331.
- the DPB 331 and the warped picture buffer 462 are a single buffer. It is possible to use both.
- the occlusion detection unit 463 detects the occlusion part of the parallax image D # 2 and supplies the occlusion information of the occlusion part to the parallax prediction unit 464, similarly to the occlusion detection unit 233 of FIG.
- the occlusion detection unit 463 is a portion having no pixel value in the picture of the warped parallax image D ′ # 1 stored in the warped picture buffer 462 (the picture at the same time as the target picture). Detect (no pixel value part).
- the occlusion detection unit 463 detects a portion of the target picture of the parallax image D # 2 at the same position as the portion without a pixel value as an occlusion portion.
- the occlusion detection unit 463 scans each block that can be processed in the target picture of the parallax image D # 2 in the horizontal direction (with parallax), thereby starting and ending the start pixel S and the end of the occlusion part. Detecting end pixel E is detected.
- the occlusion detection unit 463 When the occlusion detection unit 463 detects the start pixel S and the end pixel E of each block of the target picture, the occlusion detection unit 463 supplies the start pixel S and the end pixel E to the parallax prediction unit 464 as occlusion information.
- the disparity prediction unit 464 recognizes whether or not the target block is encoded using a prediction image generated by the disparity prediction (inter prediction) based on the header information from the variable length decoding unit 342.
- the disparity prediction unit 464 determines the reference index for prediction, that is, the target block from the header information from the variable length decoding unit 342. Recognizes (acquires) a reference index assigned to a reference picture used to generate a predicted image.
- the disparity prediction unit 464 acquires a picture of the decoded disparity image D # 1 as a reference picture that is stored in the DPB 331 and to which a reference index for prediction of the target block is assigned.
- the disparity prediction unit 234 obtains the prediction vector (of the calculated disparity vector) of the target block using the occlusion information from the occlusion detection unit 463 as necessary, similarly to the disparity prediction unit 234 of FIG. Then, the disparity prediction unit 234 adds the prediction vector and the residual vector included in the header information from the variable length decoding unit 342, so that the deviation vector used to generate the prediction image of the target block is used. Restore the calculated disparity vector.
- the disparity prediction unit 464 generates a predicted image by performing disparity prediction according to the calculated disparity vector in accordance with the calculated disparity prediction unit 234 of FIG. 13 according to the calculated disparity vector of the target block.
- the disparity prediction unit 464 predicts a block (corresponding block) at a position moved (shifted) according to the calculated disparity vector of the target block from the position of the target block of the picture of the decoded parallax image D # that is the reference picture. Obtain as an image.
- the parallax prediction unit 464 supplies the predicted image to the predicted image selection unit 451.
- a disparity prediction unit 464 that performs disparity prediction among inter predictions is provided in the decoder 322, as in the case of the encoder 22 in FIG.
- the decoder 322 performs parallax prediction and temporal prediction (prediction image generation by the same method) as in the encoder 22. .
- FIG. 24 is a block diagram illustrating a configuration example of the disparity prediction unit 464 of FIG.
- the parallax prediction unit 464 includes a prediction vector generation unit 471 and a parallax compensation unit 472.
- the prediction vector generation unit 471 is supplied with the start pixel S and the end pixel E as occlusion information from the occlusion detection unit 463.
- the prediction vector generation unit 471 uses the occlusion information from the occlusion detection unit 463 as necessary, similarly to the prediction vector generation unit 243 in FIG. 14, and uses different vector prediction methods depending on whether the target block is an occlusion block or not. Then, a prediction vector of the target block (a prediction vector in the optimal inter prediction mode) is generated.
- the prediction vector generation unit 471 when the target block is an occlusion block, the prediction vector generation unit 471 generates a prediction vector by the proximity prediction method, and when the target block is not an occlusion block, the prediction vector generation unit 471 generates a prediction vector by the MVC method. Generate.
- the prediction vector generation unit 471 supplies the prediction vector of the target block to the parallax compensation unit 472.
- the prediction vector generation unit 471 When the prediction vector generation unit 471 generates a prediction vector by the proximity prediction method, the parallax image D # 2 to be decoded by the decoder 322 is decoded as in the case of the prediction vector generation unit 243 in FIG. It is necessary to recognize whether the viewpoint # 2 is located on the left side or the right side of the viewpoint # 1 of the parallax image D # 1 serving as the reference picture.
- the recognition is performed by the variable length decoding unit 442 ( This is performed using the position information supplied to the warping unit 461 from FIG.
- the disparity compensation unit 472 is supplied with the prediction vector of the target block from the prediction vector generation unit 471, and includes the mode related information of the target block, the residual vector, and the header information from the variable length decoding unit 442, and A prediction reference index is supplied.
- the picture of the decoded parallax image D # 1, which is a reference picture stored in the DPB 331, is supplied to the parallax compensation unit 472.
- the disparity compensation unit 472 restores the calculated disparity vector mv of the target block by adding the residual vector from the variable length decoding unit 442 and the prediction vector from the prediction vector generation unit 471.
- the disparity compensation unit 472 uses, as a reference picture, a picture to which a reference index for prediction from the variable length decoding unit 442 is assigned, among the pictures of the decoded parallax image D # 1 stored in the DPB 331.
- the reference picture shift compensation (disparity compensation) is performed in the same manner as the MVC method using the calculated disparity vector mv of the target block, so that the macro block type represented by the mode-related information from the variable length decoding unit 442 Generate a predicted image of the block.
- the parallax compensation unit 472 acquires a corresponding block that is a block at a position shifted by the calculated parallax vector mv from the position of the target block in the picture of the decoded parallax image D # 1 as a predicted image.
- the parallax compensation unit 472 supplies the predicted image to the predicted image selection unit 451.
- FIG. 25 is a flowchart illustrating a decoding process for decoding the encoded data of the parallax image D # 2 of the viewpoint # 2 performed by the decoder 322 of FIG.
- step S111 the accumulation buffer 441 stores the encoded data of the parallax image D # 2 of the viewpoint # 2 supplied thereto, and the process proceeds to step S112.
- step S112 the variable length decoding unit 442 restores the quantization value and header information by reading the encoded data stored in the accumulation buffer 441 and performing variable length decoding. Then, the variable length decoding unit 442 supplies the quantized value to the inverse quantization unit 443, and supplies the header information to the intra-screen prediction unit 449 and the disparity prediction unit 450, and the process proceeds to step S113. move on.
- step S113 the inverse quantization unit 443 inversely quantizes the quantized value from the variable length decoding unit 442 into a transform coefficient, supplies the transform coefficient to the inverse orthogonal transform unit 444, and the process proceeds to step S114.
- step S114 the inverse orthogonal transform unit 444 performs inverse orthogonal transform on the transform coefficient from the inverse quantization unit 443, supplies the transform coefficient in units of macroblocks to the calculation unit 445, and the process proceeds to step S115.
- step S115 the computing unit 445 supplies the macroblock from the inverse orthogonal transform unit 444 as a target block to be decoded, and supplies the target block (residual image) from the predicted image selecting unit 451 as necessary.
- the decoded image is obtained by adding the predicted images.
- the arithmetic unit 445 supplies the decoded image to the deblocking filter 446, and the process proceeds from step S115 to step S116.
- step S116 the deblocking filter 446 performs filtering on the decoded image from the calculation unit 445, and uses the decoded image (decoded parallax image D # 2) after the filtering as the DPB 331 and the screen rearrangement buffer 447. The process proceeds to step S117.
- step S117 the DPB 331 waits for the decoded parallax image D # 1 to be supplied from the decoder 321 that decodes the parallax image D # 1, and stores the decoded parallax image D # 1, and the process proceeds to step S118. Proceed to
- step S118 the DPB 331 stores the decoded parallax image D # 2 from the deblocking filter 446, and the process proceeds to step S119.
- step S119 the warping unit 461 generates a picture of the warped parallax image D ′ # 1 by warping the picture of the decoded parallax image D # 1 stored in the DPB 331, and supplies the generated picture to the warped picture buffer 462. The process proceeds to step S120.
- step S120 the warped picture buffer 462 stores the picture of the warped parallax image D ′ # 1 from the warping unit 461, and the process proceeds to step S121.
- step S121 the occlusion detection unit 463 performs an occlusion detection process for detecting the occlusion portion of the target picture of the parallax image D # 2 using the picture of the warped parallax image D ′ # 1 stored in the warped picture buffer 462. In the same manner as in FIG.
- the occlusion detection unit 233 supplies the occlusion information obtained by the occlusion detection process to the parallax prediction unit 464, and the process proceeds from step S121 to step S122.
- step S122 the intra prediction unit 449 and the disparity prediction unit 464 perform intra prediction on the next target block (the next macro block to be decoded) based on the header information supplied from the variable length decoding unit 442. It recognizes whether it is encoded using the prediction image produced
- the intra prediction unit 449 performs an intra prediction process (intra prediction process).
- the intra-screen prediction unit 449 performs intra prediction (intra-screen prediction) for generating a predicted image (predicted image of intra prediction) from the picture of the decoded parallax image D # 2 stored in the DPB 331 for the next target block.
- the prediction image is supplied to the prediction image selection unit 451, and the process proceeds from step S122 to step S123.
- the disparity prediction unit 464 performs disparity prediction processing (inter prediction processing).
- the disparity prediction unit 464 for the next target block, the picture of the decoded disparity image D # 1 stored in the DPB 331, the mode related information included in the header information from the variable length decoding unit 442, the prediction reference index, And a prediction image is produced
- parallax prediction unit 464 supplies the predicted image to the predicted image selection unit 451, and the process proceeds from step S122 to step S123.
- step S ⁇ b> 123 the predicted image selection unit 451 selects the predicted image from the one of the intra-screen prediction unit 449 and the parallax prediction unit 464 to which the predicted image is supplied, and supplies the selected predicted image to the calculation unit 445. Then, the process proceeds to step S124.
- the predicted image selected by the predicted image selection unit 451 in step S123 is used in the process of step S115 performed in the decoding of the next target block.
- step S124 the screen rearrangement buffer 447 temporarily stores and reads the picture of the decoded parallax image D # 2 from the deblocking filter 446, so that the picture arrangement is rearranged and the D / A conversion unit. Then, the process proceeds to step S125.
- step S125 when it is necessary to output the picture from the screen rearrangement buffer 447 as an analog signal, the D / A conversion unit 348 performs D / A conversion on the picture and outputs the picture.
- FIG. 26 is a flowchart illustrating the disparity prediction process performed by the disparity prediction unit 464 in FIG. 24 in step S122 in FIG.
- step S131 the prediction vector generation unit 471 performs prediction vector generation processing for generating a prediction vector of the (next) target block using the occlusion information supplied from the occlusion detection unit 463 as necessary. This is performed as described in FIG.
- generation part 471 supplies the prediction vector of the object block obtained by the prediction vector production
- step S132 the disparity prediction unit 464 acquires the mode-related information, the residual vector, and the reference index for prediction included in the header information from the variable length decoding unit 442, and the disparity compensation The process proceeds to step S133 from step S132.
- step S132 the parallax compensation unit 472 acquires, from the DPB 331, a picture of the decoded parallax image D # 1 to which a prediction reference index is assigned (a picture at the same time as the target picture) as a reference picture, and processing is performed. The process proceeds to step S134.
- step S134 the parallax compensation unit 472 calculates the target block by adding the residual vector of the target block included in the header information from the variable length decoding unit 442 and the prediction vector from the prediction vector generation unit 471.
- the disparity vector mv is restored, and the process proceeds to step S135.
- step S135 the parallax compensation unit 472 generates a predicted image of the target block by performing MC on the decoded parallax image D # 1 picture that is a reference picture from the DPB 331 according to the calculated parallax vector mv of the target block. The process proceeds to step S136.
- the parallax compensation unit 472 acquires a corresponding block that is a block at a position shifted by the calculated parallax vector mv from the position of the target block in the picture of the decoded parallax image D # 1 as a predicted image.
- step S136 the parallax compensation unit 472 supplies the predicted image to the predicted image selection unit 451, and the process returns.
- FIG. 27 is a block diagram showing another configuration example of the encoder 22 of FIG.
- the encoder 22 of FIG. 27 includes an A / D conversion unit 211 or an intra-screen prediction unit 222, a predicted image selection unit 224, a warping unit 231, a warped picture buffer 232, and an occlusion detection unit 233. Same as the case.
- the encoder 22 of FIG. 27 is different from the case of FIG. 13 in that it includes a parallax prediction unit 534 instead of the parallax prediction unit 234.
- the prediction vector PMV of the occlusion block B # (i ⁇ 1) adjacent to the left of the target block B # i By subtracting the number of pixels E-S + 1 from the start pixel S to the end pixel E of the occlusion block from # (i-1), the prediction vector PMV # i of the target block B # i is, so to speak, in pixel units
- the prediction vector PMV # (i-1) of the occlusion block B # (i-1) adjacent to the left of the target block B # i is used in the proximity prediction method.
- the predicted vector PMV # i of the target block B # i is, so to speak, an accuracy vector in the width unit of the occlusion block. Is required
- the disparity prediction unit 534 outputs a predictor flag that is a flag representing a vector prediction method for generating a prediction vector of the target block.
- a prediction vector can be generated (without performing an occlusion detection process).
- the disparity prediction unit 534 generates a vector with accuracy in units of the width of the occlusion block as a prediction vector of the proximity prediction method, and a predictor that is a flag indicating a vector prediction method for generating a prediction vector Except for outputting a flag, processing similar to that performed by the parallax prediction unit 234 in FIG. 13 is performed.
- FIG. 28 is a block diagram illustrating a configuration example of the parallax prediction unit 534 of FIG.
- the parallax prediction unit 534 includes a parallax detection unit 541, a parallax compensation unit 542, a prediction vector generation unit 543, a cost function calculation unit 544, and a mode selection unit 545.
- the parallax detection unit 541, the parallax compensation unit 542, the cost function calculation unit 544, and the mode selection unit 545 except for handling the predictor flag, the parallax shown in FIG.
- the detection unit 241, the parallax compensation unit 242, the cost function calculation unit 244, and the mode selection unit 245 perform the same processing.
- the prediction vector generation unit 543 is the same as the prediction vector generation unit 243 in FIG. 14 except that it generates a vector with accuracy in units of the width of the occlusion block and handles a predictor flag as a prediction vector of the proximity prediction method. Perform the process.
- the picture of the decoded parallax image D # 1 that is the reference picture stored in the DPB 31 is supplied to the parallax detection unit 541, and the picture of the parallax image D # 2 to be encoded is received from the screen rearrangement buffer 212. (Target picture) is supplied.
- the disparity detection unit 541 calculates the target block by performing ME using the picture of the decoded disparity image D # 1 as a reference picture for the target block in the target picture from the screen rearrangement buffer 212, as in the MVC method.
- a disparity vector mv is detected for each macroblock type.
- the disparity detection unit 541 obtains a residual vector that is a difference between the calculated disparity vector mv of the target block and the prediction vector obtained from the prediction vector information supplied from the prediction vector generation unit 543.
- the prediction vector information supplied from the prediction vector generation unit 543 to the parallax detection unit 541 includes a prediction vector and a prediction representing a vector prediction method (here, the MVC method or the proximity prediction method) of the prediction vector.
- Device flag a vector prediction method (here, the MVC method or the proximity prediction method) of the prediction vector.
- the parallax detection unit 541 supplies the residual vector to the parallax compensation unit 542 together with the predictor flag included in the prediction vector information that represents the vector prediction method of the prediction vector used to obtain the residual vector.
- the parallax compensation unit 542 is supplied with the residual vector of the calculated parallax vector mv and the predictor flag from the parallax detection unit 541, and also with the picture of the decoded parallax image D # 1 that is the reference picture stored in the DPB 31 Is done.
- the prediction vector information is supplied from the prediction vector generation unit 543 to the parallax compensation unit 542.
- the parallax compensation unit 542 restores the calculated parallax vector mv of the target block from the residual vector from the parallax detection unit 541 and the prediction vector information from the prediction vector generation unit 543.
- the disparity compensation unit 542 uses the picture of the decoded disparity image D # 1 from the DPB 31 as a reference picture, and performs displacement compensation (disparity compensation) of the reference picture using the calculated disparity vector mv of the target block. By performing in the same manner as above, a predicted image of the target block is generated for each macroblock type.
- the parallax compensation unit 542 generates the prediction image, the residual vector from the parallax detection unit 541, the predictor flag, and the reference picture used to generate the prediction image (here, the decoded parallax image D # 1). Along with the reference index assigned to (picture), it is supplied to the cost function calculation unit 544.
- the prediction vector generation unit 543 is supplied with the start pixel S and the end pixel E as occlusion information from the occlusion detection unit 233.
- the predicted vector generation unit 543 determines whether the target block is an occlusion block using the occlusion information from the occlusion detection unit 233.
- the prediction vector generation unit 543 generates a prediction vector of the target block using the proximity prediction method described with reference to FIGS.
- the prediction vector generation unit 543 when the target block is not an occlusion block (when it is a non-occlusion block), the prediction vector generation unit 543 generates a prediction vector of the target block by the MVC method described with reference to FIGS.
- generation part 543 produces
- the prediction vector generation unit 543 generates a prediction vector for each macroblock type (FIG. 10), and includes a parallax detection unit 541 and a parallax compensation unit as prediction vector information together with a predictor flag indicating a vector prediction method of the prediction vector. 542 and the cost function calculation unit 544.
- the cost function calculation unit 544 is supplied with a prediction image, a residual vector, a predictor flag, and a reference index from the parallax compensation unit 542, and is also supplied with prediction vector information from the prediction vector generation unit 543.
- the target picture of the parallax image D # 2 is supplied from the screen rearrangement unit buffer 212.
- the cost function calculation unit 544 calculates the encoding cost required for encoding the target block of the target picture from the screen rearrangement buffer 212 by, for example, the cost function calculation unit 244 of FIG. It is determined according to a cost function similar to that used.
- the mode selection unit 545 sets the encoding cost together with the reference index, the predicted image, the residual vector, and the predictor flag. To supply.
- the mode selection unit 545 detects the minimum cost, which is the minimum value, from the encoding costs for each macroblock type from the cost function calculation unit 544.
- the mode selection unit 545 selects the macro block type for which the minimum cost is obtained as the optimal inter prediction mode.
- the mode selection unit 545 uses, as header information, mode related information representing the optimal inter prediction mode, a reference index (prediction reference index) for the optimal inter prediction mode, a residual vector, a predictor flag, and the like. And supplied to the variable length coding unit 216.
- the mode selection unit 545 supplies the prediction image and the encoding cost (minimum cost) in the optimal inter prediction mode to the prediction image selection unit 224.
- the predictor flag is included in the header of the encoded data in the variable length encoding unit 216, but the predictor flag is included in the header of the encoded data. It is possible to provide (transmit) separately from the digitized data.
- step S26 of FIG. 15 the parallax prediction unit 534 performs a parallax prediction process instead of the parallax prediction unit 234 of FIG.
- FIG. 29 is a flowchart illustrating the disparity prediction process performed by the disparity prediction unit 534 in FIG. 28 in step S26 in FIG.
- step S241 the prediction vector generation unit 543 uses the occlusion information supplied from the occlusion detection unit 233 as necessary, and calculates the prediction vector of the (next) target block for each macroblock type (FIG. 10). Prediction vector generation processing generated by the MVC method or the proximity prediction method is performed.
- the prediction vector generation unit 543 obtains a prediction vector with accuracy in units of the width of the occlusion block.
- the prediction vector generation unit 543 uses the prediction vector of the target block obtained by the prediction vector generation processing and the predictor flag representing the vector prediction method of the prediction vector as the prediction vector information, the disparity detection unit 541, the disparity The data is supplied to the compensation unit 542 and the cost function calculation unit 544, and the process proceeds from step S241 to step S242.
- step S242 the parallax prediction unit 534 acquires a picture of the decoded parallax image D # 1 that is a reference picture from the DPB 31, and supplies the decoded parallax image D # 1 to the parallax detection unit 541 and the parallax compensation unit 542, and the processing is performed in step S243. Proceed to
- step S243 the parallax detection unit 541 performs ME for the (next) target block in the target picture supplied from the screen rearrangement buffer 212, using the decoded parallax image D # 1 picture from the DPB 31 as a reference picture.
- the calculated disparity vector mv representing the disparity of the target block with respect to the viewpoint # 1 is detected.
- the disparity detection unit 541 obtains a residual vector that is a difference between the calculated disparity vector mv of the target block and the prediction vector obtained from the prediction vector information supplied from the prediction vector generation unit 543, and calculates the residual vector.
- the prediction vector is supplied to the parallax compensation unit 542 together with the predictor flag of the prediction vector used for obtaining, and the process proceeds from step S243 to step S244.
- step S244 the parallax compensation unit 542 restores the calculated parallax vector mv of the target block from the residual vector from the parallax detection unit 541 and the prediction vector information from the prediction vector generation unit 543.
- the parallax compensation unit 542 generates a predicted image of the (next) target block by performing MC according to the calculated parallax vector mv of the target block using the decoded parallax image D # 1 picture from the DPB 31 as a reference picture. To do.
- the parallax compensation unit 542 generates the prediction image, the residual vector from the parallax detection unit 541, the predictor flag, and the reference picture used to generate the prediction image (here, the decoded parallax image D #
- the reference index assigned to (picture 1) is supplied to the cost function calculation unit 544, and the process proceeds from step S244 to step S245.
- step S245 the cost function calculation unit 544 obtains, for each macroblock type (FIG. 10), the encoding cost required for encoding the target block of the target picture from the screen rearrangement buffer 212 according to a predetermined cost function.
- the reference index, predicted image, residual vector, and predictor flag from the parallax compensation unit 542 are supplied to the mode selection unit 545 together with the processing, and the process proceeds to step S246.
- step S246 the mode selection unit 545 detects the minimum cost, which is the minimum value, from the encoding costs for each macroblock type from the cost function calculation unit 544.
- the mode selection unit 545 selects the macro block type for which the minimum cost is obtained as the optimal inter prediction mode, and the process proceeds from step S246 to step S247.
- step S247 the mode selection unit 545 supplies the prediction image in the optimal inter prediction mode and the encoding cost (minimum cost) to the prediction image selection unit 224, and the process proceeds to step S248.
- step S248 the mode selection unit 545 sets the mode-related information indicating the optimal inter prediction mode, the reference index of the optimal inter prediction mode (reference index for prediction), the residual vector, and the predictor flag as header information. Is supplied to the variable length coding unit 216, and the process returns.
- FIG. 30 is a flowchart for explaining the prediction vector generation process performed by the prediction vector generation unit 543 (FIG. 28) in step S241 of FIG.
- the viewpoint # 1 is located on the right side and the viewpoint # 2 is located on the left side.
- the left side of the foreground touches.
- the prediction vector generation processing is shown when the background portion is an occlusion portion, that is, when the left side of the occlusion portion is the background and the right side is the foreground (FIG. 3).
- viewpoint # 2 of parallax image D # 2 to be encoded by encoder 22 (FIG. 27) is located on the left side of viewpoint # 1 of parallax image D # 1 serving as a reference picture Is recognized from the position information supplied to the warping unit 231 (FIG. 27).
- the prediction vector generation unit 543 performs prediction vector generation processing according to the flowchart of FIG. 30 when the viewpoint # 2 is located on the left side of the viewpoint # 1.
- step S261 the prediction vector generation unit 543 acquires the start pixel S and the end pixel E of each block of the target picture as the occlusion information from the occlusion detection unit 233 as in step S61 of FIG. The process proceeds to step S262.
- step S262 the prediction vector generation unit 543 determines whether the target block is an occlusion block based on the occlusion information from the occlusion detection unit 233, similarly to step S62 in FIG.
- step S262 If it is determined in step S262 that the target block is not an occlusion block, that is, if the target block is a non-occlusion block, the process proceeds to step S263, and the prediction vector generation unit 543 uses the MVC method The block predictive vector PMV is generated, and the process proceeds to step S264.
- step S264 the prediction vector generation unit 543 sets a value “MVC method” indicating that the prediction vector PMV is a prediction vector of the MVC method in the predictor flag, and the process returns.
- step S262 If it is determined in step S262 that the target block is an occlusion block, the process proceeds to step S265.
- the prediction vector PMV of the target block that is an occlusion block is adjacent. Generated by the prediction method.
- step S265 based on the occlusion information, the prediction vector generation unit 543 determines whether the block on the left side of the target block that is an occlusion block (encoded block) is a non-occlusion block.
- the width of the occlusion block the number of pixels in the x direction of the occlusion block
- PMV ( ⁇ BK_SIZE_X, 0)
- step S266 instead of the start pixel S and the end pixel E of the target block, the occlusion block width BK_SIZE_X is used to obtain the prediction vector PMV of the occlusion block width unit accuracy.
- step S267 the prediction vector generation unit 543 sets a value “proximity prediction method” indicating that the prediction vector PMV is a prediction vector of the proximity prediction method in the predictor flag, and the process returns.
- step S265 If it is determined in step S265 that the block on the left of the target block is not a non-occlusion block, that is, if the block on the left of the target block is an occlusion block, the process proceeds to step S268.
- the prediction vector generation unit 543 uses the width BK_SIZE_X of the occlusion block instead of the start pixel S and the end pixel E of the target block in Expression (1), and has already obtained the block (occlusion block) on the left side of the target block.
- the processing proceeds to step S267.
- step S267 the prediction vector generation unit 543 sets the value “proximity prediction method” in the predictor flag as described above, and the process returns.
- FIG. 31 is a flowchart for explaining the prediction vector generation process performed by the prediction vector generation unit 543 (FIG. 28) in step S241 of FIG.
- the viewpoint # 1 is located on the left side and the viewpoint # 2 is located on the right side, and the right side of the foreground touches in the parallax image D # 2 of the viewpoint # 2.
- the prediction vector generation processing is shown when the background portion is an occlusion portion, that is, when the right side of the occlusion portion is the background and the left side is the foreground (FIG. 4).
- viewpoint # 2 of parallax image D # 2 to be encoded by encoder 22 (FIG. 27) is positioned on the right side of viewpoint # 1 of parallax image D # 1 serving as a reference picture Is recognized from the position information supplied to the warping unit 231.
- the prediction vector generation unit 543 performs prediction vector generation processing according to the flowchart of FIG. 31 when the viewpoint # 2 is located on the right side of the viewpoint # 1.
- step S271 to step S275 the prediction vector generation unit 543 performs the same processing as that in step S261 to S265 of FIG.
- step S275 when it is determined that the block on the left of the target block is a non-occlusion block, that is, when the step (slice) of the target block in the target picture is scanned in the x direction, the process proceeds to step S276, and the prediction vector generation unit 543 determines, from the occlusion information, the number N of occlusion blocks arranged continuously in the x direction starting from the target block. Recognition (detection) is performed, and the process proceeds to step S277.
- the i-th block from the beginning of the occlusion block that is continuously arranged in the x direction starting from the target block is represented as B # i, it is continuously arranged in the x direction starting from the target block.
- the first block (target block) of the occlusion block is represented by B # 1
- the last block is represented by B # N.
- step S277 the prediction vector generation unit 543 calculates the sum N ⁇ BK_SIZE_X of the widths BK_SIZE_X of each of the N blocks B # 1 to B # N that are the occlusion blocks continuously arranged in the x direction starting from the target block.
- step S76 of FIG. 20 the prediction vector PMV # 1 with pixel-unit accuracy is obtained using the start pixel S and the end pixel E of the occlusion block.
- step S277 the start pixel S and the end pixel E are obtained. Instead, using the occlusion block width BK_SIZE_X, the prediction vector PMV # 1 with accuracy in the width unit of the occlusion block is obtained.
- step S278 the prediction vector generation unit 543 sets the value “proximity prediction method” in the predictor flag, similarly to step S267 in FIG. 30, and the process returns.
- step S275 If it is determined in step S275 that the block on the left of the target block is not a non-occlusion block, that is, if the block on the left of the target block is an occlusion block, the process proceeds to step S279.
- the prediction vector generation unit 543 replaces the start pixel S # (i-1) and the end pixel E # (i-1) of the block B # (i-1) adjacent to the left side of the target block B # i in Expression (2).
- step S278 as described above, the prediction vector generation unit 543 sets the value “proximity prediction method” in the predictor flag, and the process returns.
- FIG. 32 is a block diagram showing another configuration example of the decoder 322 of FIG.
- FIG. 32 shows a configuration example of the decoder 322 when the encoder 22 is configured as shown in FIG.
- 32 includes an accumulation buffer 441, a variable length decoding unit 442, an inverse quantization unit 443, an inverse orthogonal transform unit 444, a calculation unit 445, a deblocking filter 446, a screen rearrangement buffer 447, and a D / A conversion unit 448.
- an accumulation buffer 441 a variable length decoding unit 442, an inverse quantization unit 443, an inverse orthogonal transform unit 444, a calculation unit 445, a deblocking filter 446, a screen rearrangement buffer 447, and a D / A conversion unit 448.
- the in-screen prediction unit 449 and the predicted image selection unit 451 is common to the case of FIG.
- the decoder 322 in FIG. 32 does not include the warping unit 461, the warped picture buffer 462, and the occlusion detection unit 463, and includes a disparity prediction unit 664 instead of the disparity prediction unit 464. This is different from the case of FIG.
- the parallax prediction unit 464 of FIG. 23 obtains the prediction vector of the MVC method or the prediction vector of the proximity prediction method in the same manner as the parallax prediction unit 234 of FIG.
- the parallax prediction unit 664 in FIG. 32 obtains the accuracy vector, and based on the predictor flag, the MVC prediction vector or the proximity prediction prediction vector is obtained, and the proximity prediction prediction vector is used as the prediction vector.
- a vector of accuracy in the width unit of the occlusion block is obtained.
- parallax prediction unit 664 performs the same processing as the parallax prediction unit 464 of FIG.
- FIG. 33 is a block diagram illustrating a configuration example of the disparity prediction unit 664 of FIG.
- the parallax prediction unit 664 includes a prediction vector generation unit 771 and a parallax compensation unit 472.
- the disparity prediction unit 664 is common to the disparity prediction unit 464 of FIG. 24 in that it includes a disparity compensation unit 472, and is provided with a prediction vector generation unit 771 instead of the prediction vector generation unit 471. It is different from the parallax prediction unit 464 of FIG.
- the prediction vector generation unit 771 is supplied with the predictor flag included in the header information from the variable length decoding unit 442.
- the prediction vector generation unit 771 generates a prediction vector by the MVC method or the proximity prediction method based on the predictor flag, and supplies the prediction vector to the parallax compensation unit 472.
- the prediction vector generation unit 771 When the prediction vector generation unit 771 generates a prediction vector by the proximity prediction method, the parallax image D # 2 to be decoded by the decoder 322 is decoded as in the case of the prediction vector generation unit 543 in FIG. It is necessary to recognize whether the viewpoint # 2 is located on the left side or the right side of the viewpoint # 1 of the parallax image D # 1 serving as the reference picture. In the prediction vector generation unit 771, whether the viewpoint # 2 of the parallax image D # 2 to be decoded is positioned on the left side or the right side of the viewpoint # 1 of the parallax image D # 1 that is the reference picture is Recognition is performed based on position information included in the header information obtained by the variable length decoding unit 442 (FIG. 32).
- FIG. 34 is a flowchart for explaining a decoding process performed by the decoder 322 of FIG. 32 to decode the encoded data of the parallax image D # 2 of the viewpoint # 2.
- steps S311 to S318 the same processing as that in steps S111 to S118 of FIG. 25 is performed.
- the decoder 322 does not perform (corresponding to) the processes of steps S119 to S121 in FIG. 25, and performs the same processes as steps S122 to S125 of FIG. 25 in steps S319 to S322.
- step S319 the parallax prediction process (inter prediction process) is performed by the parallax prediction unit 664 (FIG. 33) instead of the parallax prediction unit 464 (FIG. 24).
- FIG. 35 is a flowchart for explaining the parallax prediction processing performed by the parallax prediction unit 664 in FIG. 33 in step S319 in FIG.
- step S331 the parallax prediction unit 664 acquires the predictor flag from the variable length decoding unit 442 and supplies the predictor flag to the prediction vector generation unit 771, and the process proceeds to step S332.
- step S332 the prediction vector generation unit 771 performs the prediction vector generation processing for generating the prediction vector of the (next) target block by the MVC method or the proximity prediction method based on the predictor flag, as shown in FIG. This is performed in the same manner as the vector generation unit 543.
- generation part 771 supplies the prediction vector of the object block obtained by the prediction vector production
- steps S333 to S337 the same processing as in steps S132 to S136 of FIG. 26 is performed.
- 36 and 37 are flowcharts for explaining the prediction vector generation process performed by the prediction vector generation unit 771 (FIG. 33) in step S332 of FIG.
- the viewpoint # 1 is located on the right side
- the viewpoint # 2 is located on the left side
- the parallax image D # 2 of the viewpoint # 2 When the background portion that is in contact is an occlusion portion, that is, when the left side of the occlusion portion is the background and the right side is the foreground (FIG. 3), the prediction vector generation processing in step S332 of FIG. 35 is performed. Show.
- the viewpoint # 2 of the parallax image D # 2 to be decoded is located on the left side or the right side of the viewpoint # 1 of the parallax image D # 1 serving as the reference picture Is recognized based on position information included in the header information obtained by the variable length decoding unit 442 (FIG. 32).
- step S411 the prediction vector generation unit 771 determines which of the value “MVC method” and the value “proximity prediction method” is set in the predictor flag of the target block.
- step S411 If it is determined in step S411 that the value “MVC method” is set in the predictor flag of the target block, that is, if the target block is a non-occlusion block, the process proceeds to step S412 and the prediction vector The generation unit 771 generates the prediction vector PMV of the target block using the MVC method, and the process returns.
- step S411 If it is determined in step S411 that the value “proximity prediction method” is set in the predictor flag of the target block, that is, if the target block is an occlusion block, the process proceeds to step S413.
- the prediction vector generation unit 771 sets any one of the value “MVC method” and the value “proximity prediction method” to the predictor flag of the block (decoded block) adjacent to the left of the target block that is an occlusion block. It is determined whether it is done.
- the width of the occlusion block the number of pixels in the x direction of the occlusion block
- PMV ( ⁇
- Step S413 when it is determined that the value “proximity prediction method” is set in the predictor flag of the block adjacent to the left of the target block, that is, the block adjacent to the left of the target block is an occlusion block. If there is, the process proceeds to step S415, and the prediction vector generation unit 771 has already obtained the width BK_SIZE_X of the occlusion block and the block on the left of the target block (occlusion block), as in step S268 of FIG.
- FIG. 35 shows the predicted vector generation processing in step S332 of FIG. 35 when the background portion that is in contact is an occlusion portion, that is, the right side of the occlusion portion is the background and the left side is the foreground.
- steps S421 to S423 the same processing as in steps S411 to S413 of FIG. 36 is performed.
- step S423 when it is determined that the value “MVC method” is set in the predictor flag of the block adjacent to the left of the target block, that is, the block adjacent to the left of the target block is a non-occlusion block. Therefore, when the stage (slice) of the target block of the target picture is scanned in the x direction, if the occlusion part starts from the target block, the process proceeds to step S424, and the prediction vector generation unit 771 generates a prediction vector of the proximity prediction method in steps S424 and S425 in the same manner as in steps S276 and S277 of FIG.
- step S424 based on the predictor flag, the prediction vector generation unit 771 sets an occlusion block that is continuously arranged in the x direction starting from the target block (the value “proximity prediction method” is set in the predictor flag). The number N of blocks) is recognized, and the process proceeds to step S425.
- the i-th block from the beginning of the occlusion block that is continuously arranged in the x direction starting from the target block is represented as B # i, it is continuously arranged in the x direction starting from the target block.
- the first block (target block) of the occlusion block is represented by B # 1
- the last block is represented by B # N.
- Step S423 when it is determined that the value “proximity prediction method” is set in the predictor flag of the block adjacent to the left of the target block, that is, the block adjacent to the left of the target block is an occlusion block. If there is, the process proceeds to step S426, and the prediction vector generation unit 771 determines the prediction vector PMV # (i ⁇ ) of the occlusion block width BK_SIZE_X and the block (occlusion block) to the left of the target block.
- the disparity prediction unit 664 (FIG. 33) easily generates a prediction vector with high prediction accuracy based on the predictor flag (without performing the occlusion detection process). be able to.
- a prediction vector can be generated in the same manner as the prediction vector generation unit 543 in FIG. 28 (without using a predictor flag).
- 38 to 40 are diagrams showing examples of predictor flags included in the header information.
- the predictor flag is included in the header information and supplied to the variable length coding unit 216.
- header information is included in the header of the encoded data.
- the predictor flag can be set with a macroblock as a minimum unit.
- the predictor flag is a macro block type (8 ⁇ 8 or more type) that divides the target block into an 8 ⁇ 8 pixel partition or larger, that is, a macro block type that divides the target block into an 8 ⁇ 8 pixel partition.
- (8 ⁇ 8 type) macro block type (16 ⁇ 8 type) that divides the target block into 16 ⁇ 8 pixel partitions
- the predictor flag is a macroblock type (8) that divides the target block into partitions smaller than the 8 ⁇ 8 pixel partition, that is, 8 ⁇ 4 pixel, 4 ⁇ 8 pixel, or 4 ⁇ 4 pixel subpartitions.
- a partition (subpartition) of less than ⁇ 8) can be set as the smallest unit.
- FIG. 38 is a diagram showing a predictor flag set with a macroblock as a minimum unit.
- FIG. 38 shows the syntax of MVC mb_pred (mb_type).
- the predictor flag When the predictor flag is set with a macroblock as the minimum unit, the predictor flag is included in mb_pred (mb_type).
- mv_pred_mode_l0 and mv_pred_mode_l1 indicate predictor flags.
- mv_pred_mode_l0 is a predictor flag used when L0 prediction is performed
- mv_pred_mode_l1 is a predictor flag used when L1 prediction is performed.
- FIG. 39 is a diagram showing a predictor flag set with a partition of 8 ⁇ 8 or more type as a minimum unit.
- FIG. 39 shows a partial syntax of mb_pred (mb_type) of the MVC method.
- the predictor flag When setting the predictor flag as a minimum unit of a partition of 8 ⁇ 8 or more type, the predictor flag is included in mb_pred (mb_type).
- mv_pred_mode_l0 [mbPartIdx] and mv_pred_mode_l1 [mbPartIdx] indicate predictor flags.
- mv_pred_mode_l0 [mbPartIdx] is a predictor flag used when L0 prediction is performed
- mv_pred_mode_l1 [mbPartIdx] is a predictor flag used when L1 prediction is performed.
- the argument mbPartIdx of the predictor flags mv_pred_mode_l0 [mbPartIdx] and mv_pred_mode_l1 [mbPartIdx] is an index for distinguishing each partition of 8 ⁇ 8 or more type.
- FIG. 40 is a diagram showing a predictor flag that is set with a partition of a type of less than 8 ⁇ 8 as a minimum unit.
- FIG. 40 shows a partial syntax of sub_mb_pred (mb_type) of the MVC method.
- the predictor flag When the predictor flag is set with a partition of a type less than 8 ⁇ 8 as the minimum unit, the predictor flag is included in mb_pred (mb_type) and sub_mb_pred (mb_type).
- FIG. 39 shows sub_mb_pred The predictor flag included in (mb_type) is shown.
- mv_pred_mode_l0 [mbPartIdx] [subMbPartIdx] and mv_pred_mode_l1 [mbPartIdx] [subMbPartIdx] indicate predictor flags.
- mv_pred_mode_l0 [mbPartIdx] [subMbPartIdx] is a predictor flag used when L0 prediction is performed
- mv_pred_mode_l1 [mbPartIdx] [subMbPartIdx] is used when L1 prediction is performed. Predictor flag.
- the argument subMbPartIdx of the predictor flags mv_pred_mode_l0 [mbPartIdx] [subMbPartIdx] and mv_pred_mode_l1 [mbPartIdx] [subMbPartIdx] is an index for distinguishing each partition of a type less than 8 ⁇ 8.
- the predictor flag when the predictor flag is set with a macroblock as the minimum unit, an increase in the amount of encoded data header data (overhead data amount) can be minimized.
- the predictor flag when the predictor flag is set with a partition (subpartition) of less than 8 ⁇ 8 as the minimum unit, the prediction vector can be controlled for each small-sized partition, so that the prediction accuracy is improved. be able to.
- the predictor flag when the predictor flag is set as a minimum unit of a partition of 8 ⁇ 8 or more type, a case where the macroblock is set as the minimum unit while suppressing an increase in the data amount of the header of the encoded data, and 8 ⁇ Prediction accuracy that is intermediate between the case of using a partition of less than 8 as the minimum unit can be realized.
- the MVC method (median prediction method) is adopted as a vector prediction method different from the proximity prediction method, but other vector prediction methods can be adopted as a vector prediction method different from the proximity prediction method. .
- the present technology is applied to the target block of the parallax image D # 2 for the picture of the parallax image D # 1 (another picture different from the target picture) at the same time as the target picture of the parallax image D # 2.
- the present technology is applied to a picture of a parallax image D # 2 at a time different from that of the target picture (other pictures different from the target picture).
- the present invention can also be applied to the case of obtaining a motion vector prediction vector representing the shift of the target block of the parallax image D # 2.
- the occlusion part uses the picture of the parallax image D # 1 at the same time as the target picture of the parallax image D # 2 as a reference picture and performs the parallax prediction of the target block of the parallax picture D # 2 (target picture and reference picture)
- target picture and reference picture When there is a time prediction of the target block of the parallax image D # 2 using the picture of the parallax image D # 2 at a different time from the target picture of the parallax image D # 2 as a reference picture Since this also occurs (when there is motion between the target picture and the reference picture), the present technology can also be applied to the case of obtaining a motion vector prediction vector detected by temporal prediction.
- FIG. 42 shows a configuration example of an embodiment of a computer in which a program for executing the series of processes described above is installed.
- the program can be recorded in advance in a hard disk 805 or ROM 803 as a recording medium built in the computer.
- the program can be stored (recorded) in a removable recording medium 811.
- a removable recording medium 811 can be provided as so-called package software.
- examples of the removable recording medium 811 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disc, a DVD (Digital Versatile Disc), a magnetic disc, and a semiconductor memory.
- the program can be installed in the computer from the removable recording medium 811 as described above, or can be downloaded to the computer via the communication network or the broadcast network and installed in the built-in hard disk 805. That is, the program is transferred from a download site to a computer wirelessly via a digital satellite broadcasting artificial satellite, or wired to a computer via a network such as a LAN (Local Area Network) or the Internet. be able to.
- a network such as a LAN (Local Area Network) or the Internet.
- the computer includes a CPU (Central Processing Unit) 802, and an input / output interface 810 is connected to the CPU 802 via a bus 801.
- CPU Central Processing Unit
- the CPU 802 executes a program stored in a ROM (Read Only Memory) 803 accordingly. .
- the CPU 802 loads a program stored in the hard disk 805 to a RAM (Random Access Memory) 804 and executes it.
- the CPU 802 performs processing according to the flowchart described above or processing performed by the configuration of the block diagram described above. Then, the CPU 802 outputs the processing result as necessary, for example, via the input / output interface 810, from the output unit 806, or from the communication unit 808, and further recorded on the hard disk 805.
- the input unit 807 includes a keyboard, a mouse, a microphone, and the like.
- the output unit 806 includes an LCD (Liquid Crystal Display), a speaker, and the like.
- the processing performed by the computer according to the program does not necessarily have to be performed in chronological order in the order described as the flowchart. That is, the processing performed by the computer according to the program includes processing executed in parallel or individually (for example, parallel processing or object processing).
- the program may be processed by one computer (processor), or may be distributedly processed by a plurality of computers. Furthermore, the program may be transferred to a remote computer and executed.
- FIG. 43 illustrates a schematic configuration of a television apparatus to which the present technology is applied.
- the television apparatus 900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a video signal processing unit 905, a display unit 906, an audio signal processing unit 907, a speaker 908, and an external interface unit 909. Furthermore, the television apparatus 900 includes a control unit 910, a user interface unit 911, and the like.
- the tuner 902 selects a desired channel from the broadcast wave signal received by the antenna 901, demodulates it, and outputs the obtained encoded bit stream to the demultiplexer 903.
- the demultiplexer 903 extracts video and audio packets of the program to be viewed from the encoded bit stream, and outputs the extracted packet data to the decoder 904.
- the demultiplexer 903 supplies a packet of data such as EPG (Electronic Program Guide) to the control unit 910. If scrambling is being performed, descrambling is performed by a demultiplexer or the like.
- the decoder 904 performs packet decoding processing, and outputs video data generated by the decoding processing to the video signal processing unit 905 and audio data to the audio signal processing unit 907.
- the video signal processing unit 905 performs noise removal, video processing according to user settings, and the like on the video data.
- the video signal processing unit 905 generates video data of a program to be displayed on the display unit 906, image data by processing based on an application supplied via a network, and the like.
- the video signal processing unit 905 generates video data for displaying a menu screen for selecting an item and the like, and superimposes the video data on the video data of the program.
- the video signal processing unit 905 generates a drive signal based on the video data generated in this way, and drives the display unit 906.
- the display unit 906 drives a display device (for example, a liquid crystal display element or the like) based on a drive signal from the video signal processing unit 905 to display a program video or the like.
- a display device for example, a liquid crystal display element or the like
- the audio signal processing unit 907 performs predetermined processing such as noise removal on the audio data, performs D / A conversion processing and amplification processing on the processed audio data, and outputs the audio data to the speaker 908.
- the external interface unit 909 is an interface for connecting to an external device or a network, and transmits and receives data such as video data and audio data.
- a user interface unit 911 is connected to the control unit 910.
- the user interface unit 911 includes an operation switch, a remote control signal receiving unit, and the like, and supplies an operation signal corresponding to a user operation to the control unit 910.
- the control unit 910 is configured using a CPU (Central Processing Unit), a memory, and the like.
- the memory stores a program executed by the CPU, various data necessary for the CPU to perform processing, EPG data, data acquired via a network, and the like.
- the program stored in the memory is read and executed by the CPU at a predetermined timing such as when the television device 900 is activated.
- the CPU executes the program to control each unit so that the television apparatus 900 performs an operation according to the user operation.
- the television device 900 is provided with a bus 912 for connecting the tuner 902, the demultiplexer 903, the video signal processing unit 905, the audio signal processing unit 907, the external interface unit 909, and the control unit 910.
- the decoder 904 is provided with the function of the image processing apparatus (image processing method) of the present application. For this reason, a vector with high prediction accuracy can be obtained as a prediction vector.
- FIG. 44 illustrates a schematic configuration of a mobile phone to which the present technology is applied.
- the cellular phone 920 includes a communication unit 922, an audio codec 923, a camera unit 926, an image processing unit 927, a demultiplexing unit 928, a recording / reproducing unit 929, a display unit 930, and a control unit 931. These are connected to each other via a bus 933.
- an antenna 921 is connected to the communication unit 922, and a speaker 924 and a microphone 925 are connected to the audio codec 923. Further, an operation unit 932 is connected to the control unit 931.
- the mobile phone 920 performs various operations such as transmission / reception of voice signals, transmission / reception of e-mail and image data, image shooting, and data recording in various modes such as a voice call mode and a data communication mode.
- the voice signal generated by the microphone 925 is converted into voice data and compressed by the voice codec 923 and supplied to the communication unit 922.
- the communication unit 922 performs audio data modulation processing, frequency conversion processing, and the like to generate a transmission signal.
- the communication unit 922 supplies a transmission signal to the antenna 921 and transmits it to a base station (not shown).
- the communication unit 922 performs amplification, frequency conversion processing, demodulation processing, and the like of the reception signal received by the antenna 921, and supplies the obtained audio data to the audio codec 923.
- the audio codec 923 performs data expansion of the audio data and conversion to an analog audio signal and outputs the result to the speaker 924.
- the control unit 931 receives character data input by operating the operation unit 932 and displays the input characters on the display unit 930.
- the control unit 931 generates mail data based on a user instruction or the like in the operation unit 932 and supplies the mail data to the communication unit 922.
- the communication unit 922 performs mail data modulation processing, frequency conversion processing, and the like, and transmits the obtained transmission signal from the antenna 921.
- the communication unit 922 performs amplification, frequency conversion processing, demodulation processing, and the like of the reception signal received by the antenna 921, and restores mail data. This mail data is supplied to the display unit 930 to display the mail contents.
- the mobile phone 920 can also store the received mail data in a storage medium by the recording / playback unit 929.
- the storage medium is any rewritable storage medium.
- the storage medium is a removable medium such as a semiconductor memory such as a RAM or a built-in flash memory, a hard disk, a magnetic disk, a magneto-optical disk, an optical disk, a USB memory, or a memory card.
- the image data generated by the camera unit 926 is supplied to the image processing unit 927.
- the image processing unit 927 performs encoding processing of image data and generates encoded data.
- the demultiplexing unit 928 multiplexes the encoded data generated by the image processing unit 927 and the audio data supplied from the audio codec 923 by a predetermined method, and supplies the multiplexed data to the communication unit 922.
- the communication unit 922 performs modulation processing and frequency conversion processing of multiplexed data, and transmits the obtained transmission signal from the antenna 921.
- the communication unit 922 performs amplification, frequency conversion processing, demodulation processing, and the like of the reception signal received by the antenna 921, and restores multiplexed data. This multiplexed data is supplied to the demultiplexing unit 928.
- the demultiplexing unit 928 performs demultiplexing of the multiplexed data, and supplies the encoded data to the image processing unit 927 and the audio data to the audio codec 923.
- the image processing unit 927 performs a decoding process on the encoded data to generate image data.
- the image data is supplied to the display unit 930 and the received image is displayed.
- the audio codec 923 converts the audio data into an analog audio signal, supplies the analog audio signal to the speaker 924, and outputs the received audio.
- the image processing unit 927 is provided with the function of the image processing device (image processing method) of the present application. For this reason, a vector with high prediction accuracy can be obtained as a prediction vector.
- FIG. 45 illustrates a schematic configuration of a recording / reproducing apparatus to which the present technology is applied.
- the recording / reproducing apparatus 940 records, for example, audio data and video data of a received broadcast program on a recording medium, and provides the recorded data to the user at a timing according to a user instruction.
- the recording / reproducing device 940 can also acquire audio data and video data from another device, for example, and record them on a recording medium. Further, the recording / reproducing apparatus 940 decodes and outputs the audio data and video data recorded on the recording medium, thereby enabling image display and audio output on the monitor apparatus or the like.
- the recording / reproducing apparatus 940 includes a tuner 941, an external interface unit 942, an encoder 943, an HDD (Hard Disk Drive) unit 944, a disk drive 945, a selector 946, a decoder 947, an OSD (On-Screen Display) unit 948, a control unit 949, A user interface unit 950 is included.
- Tuner 941 selects a desired channel from a broadcast signal received by an antenna (not shown).
- the tuner 941 outputs an encoded bit stream obtained by demodulating the received signal of a desired channel to the selector 946.
- the external interface unit 942 includes at least one of an IEEE 1394 interface, a network interface unit, a USB interface, a flash memory interface, and the like.
- the external interface unit 942 is an interface for connecting to an external device, a network, a memory card, and the like, and receives data such as video data and audio data to be recorded.
- the encoder 943 performs encoding by a predetermined method when the video data and audio data supplied from the external interface unit 942 are not encoded, and outputs an encoded bit stream to the selector 946.
- the HDD unit 944 records content data such as video and audio, various programs, and other data on a built-in hard disk, and reads them from the hard disk during playback.
- the disk drive 945 records and reproduces signals with respect to the mounted optical disk.
- An optical disk such as a DVD disk (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD + R, DVD + RW, etc.), Blu-ray disk, or the like.
- the selector 946 selects one of the encoded bit streams from the tuner 941 or the encoder 943 and supplies it to either the HDD unit 944 or the disk drive 945 when recording video or audio. Further, the selector 946 supplies the encoded bit stream output from the HDD unit 944 or the disk drive 945 to the decoder 947 at the time of reproduction of video and audio.
- the decoder 947 performs a decoding process on the encoded bit stream.
- the decoder 947 supplies the video data generated by performing the decoding process to the OSD unit 948.
- the decoder 947 outputs audio data generated by performing the decoding process.
- the OSD unit 948 generates video data for displaying a menu screen for selecting an item and the like, and superimposes it on the video data output from the decoder 947 and outputs the video data.
- a user interface unit 950 is connected to the control unit 949.
- the user interface unit 950 includes an operation switch, a remote control signal receiving unit, and the like, and supplies an operation signal corresponding to a user operation to the control unit 949.
- the control unit 949 is configured using a CPU, a memory, and the like.
- the memory stores programs executed by the CPU and various data necessary for the CPU to perform processing.
- the program stored in the memory is read and executed by the CPU at a predetermined timing such as when the recording / reproducing apparatus 940 is activated.
- the CPU executes the program to control each unit so that the recording / reproducing device 940 operates according to the user operation.
- the decoder 947 is provided with the function of the image processing apparatus (image processing method) of the present application. For this reason, a vector with high prediction accuracy can be obtained as a prediction vector.
- FIG. 46 illustrates a schematic configuration of an imaging apparatus to which the present technology is applied.
- the imaging device 960 images a subject, displays an image of the subject on a display unit, and records it on a recording medium as image data.
- the imaging device 960 includes an optical block 961, an imaging unit 962, a camera signal processing unit 963, an image data processing unit 964, a display unit 965, an external interface unit 966, a memory unit 967, a media drive 968, an OSD unit 969, and a control unit 970. Have. In addition, a user interface unit 971 is connected to the control unit 970. Furthermore, the image data processing unit 964, the external interface unit 966, the memory unit 967, the media drive 968, the OSD unit 969, the control unit 970, and the like are connected via a bus 972.
- the optical block 961 is configured using a focus lens, a diaphragm mechanism, and the like.
- the optical block 961 forms an optical image of the subject on the imaging surface of the imaging unit 962.
- the imaging unit 962 is configured using a CCD or CMOS image sensor, generates an electrical signal corresponding to the optical image by photoelectric conversion, and supplies the electrical signal to the camera signal processing unit 963.
- the camera signal processing unit 963 performs various camera signal processing such as knee correction, gamma correction, and color correction on the electrical signal supplied from the imaging unit 962.
- the camera signal processing unit 963 supplies the image data after the camera signal processing to the image data processing unit 964.
- the image data processing unit 964 performs an encoding process on the image data supplied from the camera signal processing unit 963.
- the image data processing unit 964 supplies the encoded data generated by performing the encoding process to the external interface unit 966 and the media drive 968. Further, the image data processing unit 964 performs a decoding process on the encoded data supplied from the external interface unit 966 and the media drive 968.
- the image data processing unit 964 supplies the image data generated by performing the decoding process to the display unit 965. Further, the image data processing unit 964 superimposes the processing for supplying the image data supplied from the camera signal processing unit 963 to the display unit 965 and the display data acquired from the OSD unit 969 on the image data. To supply.
- the OSD unit 969 generates display data such as a menu screen and icons made up of symbols, characters, or figures and outputs them to the image data processing unit 964.
- the external interface unit 966 includes, for example, a USB input / output terminal, and is connected to a printer when printing an image.
- a drive is connected to the external interface unit 966 as necessary, a removable medium such as a magnetic disk or an optical disk is appropriately mounted, and a computer program read from them is installed as necessary.
- the external interface unit 966 has a network interface connected to a predetermined network such as a LAN or the Internet.
- the control unit 970 reads the encoded data from the memory unit 967 in accordance with an instruction from the user interface unit 971, and supplies the encoded data to the other device connected via the network from the external interface unit 966. it can.
- the control unit 970 may acquire encoded data and image data supplied from another device via the network via the external interface unit 966 and supply the acquired data to the image data processing unit 964. it can.
- any readable / writable removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory is used.
- the recording medium may be any type of removable medium, and may be a tape device, a disk, or a memory card. Of course, a non-contact IC card or the like may be used.
- media drive 968 and the recording medium may be integrated and configured by a non-portable storage medium such as a built-in hard disk drive or an SSD (Solid State Drive).
- a non-portable storage medium such as a built-in hard disk drive or an SSD (Solid State Drive).
- the control unit 970 is configured using a CPU, a memory, and the like.
- the memory stores programs executed by the CPU, various data necessary for the CPU to perform processing, and the like.
- the program stored in the memory is read and executed by the CPU at a predetermined timing such as when the imaging device 960 is activated.
- the CPU executes the program to control each unit so that the imaging device 960 operates according to the user operation.
- the image data processing unit 964 is provided with the function of the image processing apparatus (image processing method) of the present application. For this reason, a vector with high prediction accuracy can be obtained as a prediction vector.
- the present technology is not limited to encoding and decoding using MVC. That is, the present technology can be applied to the case where encoding and decoding of a disparity information image are performed using a prediction vector.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
Description
図41は、視差と奥行きについて説明する図である。
・・・(1)
・・・(2)
・・・(3)
・・・(4)
・・・(5)
・・・(6)
また、ブロックB#2の開始画素S#2と終了画素E#2は、それぞれ、1画素目と16画素目になり、ブロックB#3の開始画素S#3と終了画素E#3は、それぞれ、1画素目と12画素目になる。
図43は、本技術を適用したテレビジョン装置の概略構成を例示している。テレビジョン装置900は、アンテナ901、チューナ902、デマルチプレクサ903、デコーダ904、映像信号処理部905、表示部906、音声信号処理部907、スピーカ908、外部インタフェース部909を有している。さらに、テレビジョン装置900は、制御部910、ユーザインタフェース部911等を有している。
図44は、本技術を適用した携帯電話機の概略構成を例示している。携帯電話機920は、通信部922、音声コーデック923、カメラ部926、画像処理部927、多重分離部928、記録再生部929、表示部930、制御部931を有している。これらは、バス933を介して互いに接続されている。
図45は、本技術を適用した記録再生装置の概略構成を例示している。記録再生装置940は、例えば受信した放送番組のオーディオデータとビデオデータを、記録媒体に記録して、その記録されたデータをユーザの指示に応じたタイミングでユーザに提供する。また、記録再生装置940は、例えば他の装置からオーディオデータやビデオデータを取得し、それらを記録媒体に記録させることもできる。さらに、記録再生装置940は、記録媒体に記録されているオーディオデータやビデオデータを復号して出力することで、モニタ装置等において画像表示や音声出力を行うことができるようにする。
図46は、本技術を適用した撮像装置の概略構成を例示している。撮像装置960は、被写体を撮像し、被写体の画像を表示部に表示させたり、それを画像データとして、記録媒体に記録する。
Claims (12)
- 色画像の画素ごとの視差に関するデプス情報を画素値として有するデプス画像の処理対象の対象ブロックの、前記対象ブロックのピクチャと異なる他のピクチャに対するずれを表すずれベクトルの予測ベクトルを、前記対象ブロックが、前記他のピクチャに対応点が存在しないオクルージョン部分を含むオクルージョンブロックであるかどうかによって、異なるベクトル予測方式で生成する予測ベクトル生成部を備える
画像処理装置。 - 前記予測ベクトル生成部は、第1の視点と異なる第2の視点の色画像の画素ごとの視差に関するデプス情報を画素値として有する第2の視点のデプス画像の前記対象ブロックの、前記第1の視点のデプス画像のピクチャに対するずれを表すずれベクトルである視差ベクトルの予測ベクトルを、前記対象ブロックが、前記第1の視点のデプス画像のピクチャに対応点が存在しないオクルージョン部分を含むオクルージョンブロックであるかどうかによって、異なるベクトル予測方式で生成する
請求項1に記載の画像処理装置。 - 前記第1の視点のデプス画像のピクチャをワーピングすることにより、前記第1の視点のデプス画像のピクチャを前記第2の視点で得られる画像に変換したワープドデプス画像のピクチャを生成するワーピング部と、
前記ワープドデプス画像のピクチャを用いて、前記対象ブロックのピクチャである対象ピクチャのオクルージョン部分を検出するオクルージョン検出部と
をさらに備える請求項2に記載の画像処理装置。 - 前記予測ベクトル生成部は、前記対象ブロックが、前記オクルージョンブロックである場合、前記対象ブロックから、前記第1の視点のデプス画像のピクチャの、前記オクルージョン部分と同一位置のオクルージョン対応部分の外部側に近接する近接部分へのベクトルを、前記対象ブロックの予測ベクトルとして生成する近接予測方式で、前記対象ブロックの予測ベクトルを生成する
請求項3に記載の画像処理装置。 - 前記オクルージョン検出部は、前記対象ピクチャにおいて前記処理対象となりうる各ブロックについて、水平方向にスキャンすることにより、オクルージョン部分が開始している開始画素と終了している終了画素とを検出し、
前記予測ベクトル生成部は、前記近接予測方式において、前記対象ブロックの左隣のオクルージョンブロックの予測ベクトルから、前記オクルージョンブロックの前記開始画素から前記終了画素までの画素数を減算することにより求められる、前記対象ブロックから、前記オクルージョン対応部分の境界の外部側に接する前記近接部分へのベクトルを、前記対象ブロックの予測ベクトルとして生成する
請求項4に記載の画像処理装置。 - 前記予測ベクトル生成部は、前記対象ブロックの左隣のオクルージョンブロックの予測ベクトルから、前記対象ブロックの前記開始画素から前記終了画素までの画素数を減算することにより求められる、前記対象ブロックから、前記オクルージョン対応部分の境界の外部側に接する前記近接部分へのベクトルを、前記対象ブロックの予測ベクトルとして生成する
請求項5に記載の画像処理装置。 - 前記予測ベクトル生成部は、前記近接予測方式において、前記対象ブロックの左隣のオクルージョンブロックの予測ベクトルから、前記オクルージョンブロックの水平方向の画素数を減算することにより求められる、前記対象ブロックから、前記オクルージョン対応部分の外部側に近接する前記近接部分へのベクトルを、前記対象ブロックの予測ベクトルとして生成する
請求項4に記載の画像処理装置。 - 前記予測ベクトル生成部は、前記対象ブロックの予測ベクトルのベクトル予測方式を表す予測器フラグを出力する
請求項7に記載の画像処理装置。 - 前記予測ベクトル生成部は、前記対象ブロックの予測ベクトルのベクトル予測方式を表す予測器フラグを取得し、その予測器フラグが表すベクトル予測方式で、前記対象ブロックの予測ベクトルを生成する
請求項4に記載の画像処理装置。 - 前記予測ベクトル生成部は、前記予測器フラグが、前記近接予測方式を表している場合、前記対象ブロックの左隣のオクルージョンブロックの予測ベクトルから、前記オクルージョンブロックの水平方向の画素数を減算することにより求められる、前記対象ブロックから、前記オクルージョン対応部分の外部側に近接する前記近接部分へのベクトルを、前記対象ブロックの予測ベクトルとして生成する
請求項9に記載の画像処理装置。 - 色画像の画素ごとの視差に関するデプス情報を画素値として有するデプス画像の処理対象の対象ブロックの、前記対象ブロックのピクチャと異なる他のピクチャに対するずれを表すずれベクトルの予測ベクトルを、前記対象ブロックが、前記他のピクチャに対応点が存在しないオクルージョン部分を含むオクルージョンブロックであるかどうかによって、異なるベクトル予測方式で生成する
ステップを含む画像処理方法。 - 色画像の画素ごとの視差に関するデプス情報を画素値として有するデプス画像の処理対象の対象ブロックの、前記対象ブロックのピクチャと異なる他のピクチャに対するずれを表すずれベクトルの予測ベクトルを、前記対象ブロックが、前記他のピクチャに対応点が存在しないオクルージョン部分を含むオクルージョンブロックであるかどうかによって、異なるベクトル予測方式で生成する予測ベクトル生成部
として、コンピュータを機能させるためのプログラム。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/004,285 US9445092B2 (en) | 2011-03-18 | 2012-03-19 | Image processing apparatus, image processing method, and program |
JP2013505962A JPWO2012128242A1 (ja) | 2011-03-18 | 2012-03-19 | 画像処理装置、画像処理方法、及び、プログラム |
CN2012800127355A CN103416064A (zh) | 2011-03-18 | 2012-03-19 | 图像处理设备、图像处理方法和程序 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2011061794 | 2011-03-18 | ||
JP2011-061794 | 2011-03-18 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2012128242A1 true WO2012128242A1 (ja) | 2012-09-27 |
Family
ID=46879389
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2012/056975 WO2012128242A1 (ja) | 2011-03-18 | 2012-03-19 | 画像処理装置、画像処理方法、及び、プログラム |
Country Status (4)
Country | Link |
---|---|
US (1) | US9445092B2 (ja) |
JP (1) | JPWO2012128242A1 (ja) |
CN (1) | CN103416064A (ja) |
WO (1) | WO2012128242A1 (ja) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2014075687A (ja) * | 2012-10-04 | 2014-04-24 | Seiko Epson Corp | 画像データ圧縮装置、画像データ伸張装置、表示装置、画像処理システム、画像データ圧縮方法及び画像データ伸張方法 |
JP2016537839A (ja) * | 2013-09-26 | 2016-12-01 | クゥアルコム・インコーポレイテッドQualcomm Incorporated | Hevcにおけるサブ予測ユニット(pu)ベースの時間的動きベクトル予測および3d−hevcにおけるサブpu設計 |
Families Citing this family (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3247116A1 (en) * | 2012-07-27 | 2017-11-22 | HFI Innovation Inc. | Method of constrain disparity vector derivation in 3d video coding |
EP3061233B1 (en) | 2013-10-25 | 2019-12-11 | Microsoft Technology Licensing, LLC | Representing blocks with hash values in video and image coding and decoding |
EP3061253A1 (en) | 2013-10-25 | 2016-08-31 | Microsoft Technology Licensing, LLC | Hash-based block matching in video and image coding |
WO2015131325A1 (en) | 2014-03-04 | 2015-09-11 | Microsoft Technology Licensing, Llc | Hash table construction and availability checking for hash-based block matching |
US10368092B2 (en) * | 2014-03-04 | 2019-07-30 | Microsoft Technology Licensing, Llc | Encoder-side decisions for block flipping and skip mode in intra block copy prediction |
CN105706450B (zh) | 2014-06-23 | 2019-07-16 | 微软技术许可有限责任公司 | 根据基于散列的块匹配的结果的编码器决定 |
JP6365153B2 (ja) * | 2014-09-10 | 2018-08-01 | 株式会社ソシオネクスト | 画像符号化方法および画像符号化装置 |
KR102358276B1 (ko) | 2014-09-30 | 2022-02-04 | 마이크로소프트 테크놀로지 라이센싱, 엘엘씨 | 비디오 코딩을 위한 해시 기반 인코더 결정 |
JP2017069617A (ja) * | 2015-09-28 | 2017-04-06 | ルネサスエレクトロニクス株式会社 | 半導体装置および画像符号化方法 |
DE102017103588A1 (de) * | 2016-02-24 | 2017-08-24 | Jtekt Corporation | Analysevorrichtung und analysesystem |
US10390039B2 (en) | 2016-08-31 | 2019-08-20 | Microsoft Technology Licensing, Llc | Motion estimation for screen remoting scenarios |
US11095877B2 (en) | 2016-11-30 | 2021-08-17 | Microsoft Technology Licensing, Llc | Local hash-based motion estimation for screen remoting scenarios |
US10586308B2 (en) * | 2017-05-09 | 2020-03-10 | Adobe Inc. | Digital media environment for removal of obstructions in a digital image scene |
CN109005355B (zh) * | 2018-09-28 | 2020-10-09 | 维沃移动通信有限公司 | 一种拍摄方法及移动终端 |
US11202085B1 (en) | 2020-06-12 | 2021-12-14 | Microsoft Technology Licensing, Llc | Low-cost hash table construction and hash-based block matching for variable-size blocks |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2004227231A (ja) * | 2003-01-22 | 2004-08-12 | Nippon Telegr & Teleph Corp <Ntt> | 画像データへの立体情報埋め込み方法、装置、プログラム、および記録媒体 |
JP2006513661A (ja) * | 2003-01-23 | 2006-04-20 | コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ | バックグラウンド動きベクトル検出方法及び装置 |
JP2010154490A (ja) * | 2008-11-26 | 2010-07-08 | Hitachi Ltd | 画像復号化方法 |
JP2010157826A (ja) * | 2008-12-26 | 2010-07-15 | Victor Co Of Japan Ltd | 画像復号装置、画像符復号方法およびそのプログラム |
Family Cites Families (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8823821B2 (en) * | 2004-12-17 | 2014-09-02 | Mitsubishi Electric Research Laboratories, Inc. | Method and system for processing multiview videos for view synthesis using motion vector predictor list |
US7596243B2 (en) * | 2005-09-16 | 2009-09-29 | Sony Corporation | Extracting a moving object boundary |
ZA200805337B (en) * | 2006-01-09 | 2009-11-25 | Thomson Licensing | Method and apparatus for providing reduced resolution update mode for multiview video coding |
CA2672924C (en) * | 2006-12-28 | 2014-07-22 | Nippon Telegraph And Telephone Corporation | Video encoding method and decoding method, apparatuses therefor, programs therefor, and storage media which store the programs |
WO2010073513A1 (ja) * | 2008-12-26 | 2010-07-01 | 日本ビクター株式会社 | 画像符号化装置、画像符号化方法およびそのプログラム、ならびに画像復号装置、画像復号方法およびそのプログラム |
KR101628383B1 (ko) * | 2010-02-26 | 2016-06-21 | 연세대학교 산학협력단 | 영상 처리 장치 및 방법 |
JP6061150B2 (ja) * | 2011-03-18 | 2017-01-18 | ソニー株式会社 | 画像処理装置、画像処理方法、及び、プログラム |
BR112014004062A2 (pt) * | 2011-08-31 | 2017-03-07 | Sony Corp | dispositivos e métodos de codificação e decodificação |
WO2013031575A1 (ja) * | 2011-08-31 | 2013-03-07 | ソニー株式会社 | 画像処理装置および画像処理方法 |
JP2013207755A (ja) * | 2012-03-29 | 2013-10-07 | Sony Corp | 画像処理装置および方法 |
-
2012
- 2012-03-19 JP JP2013505962A patent/JPWO2012128242A1/ja active Pending
- 2012-03-19 CN CN2012800127355A patent/CN103416064A/zh active Pending
- 2012-03-19 WO PCT/JP2012/056975 patent/WO2012128242A1/ja active Application Filing
- 2012-03-19 US US14/004,285 patent/US9445092B2/en not_active Expired - Fee Related
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2004227231A (ja) * | 2003-01-22 | 2004-08-12 | Nippon Telegr & Teleph Corp <Ntt> | 画像データへの立体情報埋め込み方法、装置、プログラム、および記録媒体 |
JP2006513661A (ja) * | 2003-01-23 | 2006-04-20 | コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ | バックグラウンド動きベクトル検出方法及び装置 |
JP2010154490A (ja) * | 2008-11-26 | 2010-07-08 | Hitachi Ltd | 画像復号化方法 |
JP2010157826A (ja) * | 2008-12-26 | 2010-07-15 | Victor Co Of Japan Ltd | 画像復号装置、画像符復号方法およびそのプログラム |
Non-Patent Citations (1)
Title |
---|
TAKANORI SENOO ET AL.: "Tashiten Eizo no Shisa Hosho Yosoku no Kento", THE JOURNAL OF THE INSTITUTE OF IMAGE ELECTRONICS ENGINEERS OF JAPAN, THE INSTITUTE OF IMAGE ELECTRONICS ENGINEERS OF JAPAN, vol. 35, no. 5, 25 September 2006 (2006-09-25), pages 488 - 496 * |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2014075687A (ja) * | 2012-10-04 | 2014-04-24 | Seiko Epson Corp | 画像データ圧縮装置、画像データ伸張装置、表示装置、画像処理システム、画像データ圧縮方法及び画像データ伸張方法 |
JP2016537839A (ja) * | 2013-09-26 | 2016-12-01 | クゥアルコム・インコーポレイテッドQualcomm Incorporated | Hevcにおけるサブ予測ユニット(pu)ベースの時間的動きベクトル予測および3d−hevcにおけるサブpu設計 |
Also Published As
Publication number | Publication date |
---|---|
JPWO2012128242A1 (ja) | 2014-07-24 |
US9445092B2 (en) | 2016-09-13 |
US20140002603A1 (en) | 2014-01-02 |
CN103416064A (zh) | 2013-11-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2012128242A1 (ja) | 画像処理装置、画像処理方法、及び、プログラム | |
JP6061150B2 (ja) | 画像処理装置、画像処理方法、及び、プログラム | |
EP3047649B1 (en) | Combined bi-predictive merging candidates for 3d video coding | |
WO2012121052A1 (ja) | 画像処理装置、画像処理方法、及び、プログラム | |
EP3092805B1 (en) | Block-based advanced residual prediction for 3d video coding | |
EP3011746B1 (en) | More accurate advanced residual prediction (arp) for texture coding | |
WO2012147622A1 (ja) | 画像処理装置、及び、画像処理方法 | |
US9350972B2 (en) | Encoding device and encoding method, and decoding device and decoding method | |
WO2012157443A1 (ja) | 画像処理装置、及び、画像処理方法 | |
KR101918324B1 (ko) | 화상 처리 장치 및 방법 | |
WO2014100610A1 (en) | Constraints on neighboring block based disparity vector (nbdv) techniques for 3d video | |
JPWO2016104179A1 (ja) | 画像処理装置および画像処理方法 | |
WO2012128241A1 (ja) | 画像処理装置、画像処理方法、及び、プログラム | |
WO2013157439A1 (ja) | 復号装置および復号方法、並びに、符号化装置および符号化方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 12759977 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2013505962 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 14004285 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 12759977 Country of ref document: EP Kind code of ref document: A1 |