JP5759357B2 - Video encoding method, video decoding method, video encoding device, video decoding device, video encoding program, and video decoding program - Google Patents

Description

  The present invention relates to a video encoding method, a video decoding method, a video encoding device, a video decoding device, a video encoding program, and a video decoding program.

  A free viewpoint video is a video that allows the user to freely specify the position and orientation of the camera in the shooting space (hereinafter referred to as the viewpoint). In a free viewpoint video, since the user designates an arbitrary viewpoint, it is not realistic to hold the video for all the possibilities, so depending on the information group necessary to generate the video of the designated viewpoint Composed. Note that the free viewpoint video may also be referred to as a free viewpoint television, an arbitrary viewpoint video, an arbitrary viewpoint television, or the like.

  A free viewpoint video is expressed using various data formats. As a most general format, there is a method using a video and a depth map (distance image) for each frame of the video (for example, see Non-Patent Document 1). . Here, the depth map represents the depth (distance) from the shooting position of the camera to the subject for each pixel, and represents the three-dimensional position of the subject. Since the depth is proportional to the reciprocal of the parallax between the two cameras, it is sometimes called a disparity map (parallax image). In the field of computer graphics, the depth is information stored in the Z buffer, so it is sometimes called a Z image or a Z map. In addition to the distance from the camera to the subject, a coordinate value with respect to the Z axis of the three-dimensional coordinate system stretched on the expression target space may be used as the depth. In general, since the horizontal direction is the X axis and the vertical direction is the Y axis with respect to the captured image, the Z axis coincides with the direction of the camera, but when a common coordinate system is used for a plurality of cameras, etc. In some cases, the Z-axis does not match the camera orientation. Hereinafter, the distance and the Z value are referred to as depth without distinction, and an image representing the depth as a pixel value is referred to as a depth map. However, strictly speaking, it is necessary to set a reference camera pair in the disparity map.

  When expressing the depth as a pixel value, the value corresponding to the physical quantity is directly used as the pixel value, the method using a value obtained by quantizing the value between the minimum value and the maximum value into a certain number, and the difference from the minimum value. There is a method of using a value obtained by quantizing with a step width. When the range to be expressed is limited, the depth can be expressed with higher accuracy by using additional information such as a minimum value. In addition, when quantizing at equal intervals, there are a method of quantizing a physical quantity as it is and a method of quantizing an inverse of a physical quantity. Since the reciprocal of the distance is a value proportional to the parallax, the former is used when the distance needs to be expressed with high accuracy, and the latter is used when the parallax needs to be expressed with high accuracy. Often. In the following description, everything in which depth is expressed as an image is referred to as a depth map regardless of the pixel value conversion method or the quantization method.

  The depth map can be regarded as a grayscale image because each pixel is expressed as an image having one value. In addition, since the subject exists continuously in the real space and cannot move to a position distant from the moment, it can be said that the subject has a spatial correlation and a temporal correlation like the image signal. Therefore, depending on the image coding method and video coding method used to encode normal image signals and video signals, images composed of depth maps and continuous depth maps can be spatially and temporally redundant. It is possible to efficiently encode while removing. Hereinafter, the depth map and its video are referred to as a depth map without distinction.

  Here, general video coding will be described. In video coding, in order to realize efficient coding using the feature that the subject is spatially and temporally continuous, each frame of the video is divided into processing unit blocks called macroblocks, The video signal is predicted spatially or temporally for each macroblock, and prediction information indicating the prediction method and a prediction residual are encoded. When predicting a video signal spatially, for example, information indicating the direction of spatial prediction becomes prediction information, and when predicting temporally, for example, information indicating a frame to be referenced and information indicating a position in the frame Is prediction information.

  Since the prediction performed spatially is an intra-frame prediction, it is called intra-frame prediction (intra-screen prediction, intra prediction). H. In intra-frame prediction used in recent video coding systems represented by H.264 / AVC, spatial continuity with adjacent blocks in an image in the block, such as texture continuity with adjacent blocks, for a prediction target block. Information indicating the direction of correlation is used. Specifically, one direction is selected from a plurality of types of directions prepared in advance for the block to be predicted, and already encoded (or decoded) adjacent to the block to be predicted according to the direction. A predicted image is generated using the pixel value of the pixel of (completed). By finely setting the selectable prediction directions, the correlation can be used with higher accuracy. H. In H.264 / AVC, depending on the size of a block to be predicted, up to eight types of directions can be used. (For details of H.264 / AVC, see Non-Patent Document 2, for example).

  By making it possible to finely set the prediction direction of intra-frame prediction, prediction corresponding to the continuity of textures in various directions becomes possible. However, if the prediction direction can be set finely, it is impossible to perform correct intraframe prediction when different subjects are captured in the prediction target block and its adjacent pixels due to occlusion or the like.

  With respect to such a problem, in Patent Document 1, in addition to the prediction direction, a distance from the prediction target block is specified, and generation of a predicted image using pixels other than adjacent pixels causes occlusion. In the case of, correct intra prediction is realized. By using this method, it is possible to perform intra-frame prediction with reference to distant pixels other than adjacent pixels. Therefore, there are occlusions and noise at the block end, and spatially periodic signals exist. In some cases, the video signal can be predicted with high accuracy.

Republished WO2008 / 102805

Y. Mori, N. Fukushima, T. Fujii, and M. Tanimoto, “View Generation with 3D Warping Using Depth Information for FTV”, In Proceedings of 3DTV-CON2008, pp. 229-232, May 2008. Rec. ITU-T H.264, “Advanced video coding for generic audiovisual services”, March 2009.

  However, in the method of Patent Document 1, in addition to the prediction direction, it is necessary to encode information indicating which pixel is used as a reference pixel for each block. As a result, since the code amount of the additional information in the block that performs intra-frame prediction increases, there is a problem that efficient coding cannot be realized.

  Moreover, since the method of patent document 1 defines the distance from one prediction object block for every prediction object block, and designates a reference pixel using the same distance with respect to all the pixels in a prediction object block, When the subject shape is not parallel to the block boundary, or when there are a plurality of subjects in the prediction direction, there is a problem that prediction with high accuracy cannot be performed. It is also possible to use a method of designating a distance for indicating a reference pixel for each pixel in the block, but this method indicates which pixel is used as a reference pixel for each pixel. There is a problem that the information must be encoded, and the amount of additional information is further increased, so that efficient encoding cannot be realized.

  The present invention has been made in view of such circumstances, and a video code capable of improving the encoding efficiency of intra-frame prediction in encoding of free-viewpoint video data having a video and a depth map as components. It is an object to provide an encoding method, a video decoding method, a video encoding device, a video decoding device, a video encoding program, and a video decoding program.

  The present invention is a video in which predictive encoding is performed using intra-frame prediction on a processing region obtained by dividing each frame constituting the video into a predetermined size while using a depth map corresponding to the video. An encoding method, an intra-frame prediction direction setting step for setting a prediction direction in the intra-frame prediction, and the processing region for each pixel or a predetermined number of pixels in the processing region according to the set prediction direction A reference pixel candidate setting step for setting a pixel within a predetermined distance range from the reference region as a reference pixel candidate, a depth map corresponding to the processing region, and a depth map corresponding to the reference pixel candidate. A reference pixel dissimilarity calculating step for calculating a dissimilarity for each candidate of the reference pixel for each pixel in the region or a predetermined number of pixels, and the difference A reference pixel determining step for determining a reference pixel from among the reference pixel candidates for each pixel or a predetermined number of pixels in the processing region, and according to the set prediction direction, And a predicted image generation step for generating a predicted image for the processing region.

  ADVANTAGE OF THE INVENTION According to this invention, the code amount required for the encoding of a prediction residual with the improvement of the prediction precision of a video signal can be reduced. Occlusion at the edge of the region is enabled not only by the pixels adjacent to the prediction target region in intraframe prediction, but also by prediction from pixels located within a certain distance and along the prediction direction of intraframe prediction. In addition, pixel values from the same subject can be predicted even in a case where noise is included or in a region where there are a plurality of subjects in the prediction direction, so that highly accurate prediction can be realized. In the conventional method (for example, Patent Document 1), the distance from the prediction target region is specified, and intra-frame prediction using pixels other than the pixels adjacent to the prediction target region is realized. In this case, since it is necessary to encode information for specifying the distance, the distance cannot be specified for each pixel, and high-precision prediction cannot be performed for all the pixels. In the present invention, using the depth map information transmitted separately, a pixel in which the same subject as the pixel to be predicted is captured is selected from the reference pixel candidates, and the selected reference pixel is used. To generate a predicted image. In this way, it is not necessary to encode information specifying the distance. As a result, it is possible to determine the distance for each pixel, and it is possible to realize highly accurate prediction for all the pixels. Here, since the value of the depth map is largely dependent on the subject, it is possible to determine whether or not the same subject is captured with high accuracy by determining the subject using the depth map. Even when the distance is designated for each pixel group, it is not necessary to encode the information for designating the distance, which is necessary in Patent Document 1, so that the amount of code can be reduced.

  In the present invention, the reference pixel dissimilarity calculating step includes a difference value between a value of the depth map corresponding to a pixel in the processing region or a predetermined number of pixel groups and a value of the depth map corresponding to the reference pixel candidate. And the distance between the processing region and the reference pixel is used to calculate the degree of difference for each candidate of the reference pixel for each pixel in the processing region or a predetermined number of pixel groups.

  According to the present invention, it is possible to achieve a reduction in the amount of code necessary for encoding a prediction residual accompanying an improvement in prediction accuracy by performing prediction from a pixel having a higher correlation with respect to a prediction target pixel. . Since the texture information is lost in the depth map, it is not possible to determine the reference pixel taking into account the texture in the same subject. In general, the closer the spatial distance in the image, the higher the correlation between the pixel values. Therefore, in addition to the depth map value, the reference pixel is determined based on the distance standard between the reference pixel and the prediction target pixel. When there are a plurality of pixels in which the same subject appears in the reference image candidates, prediction using pixels with higher correlation becomes possible.

  In the present invention, in the reference pixel determining step, the weighted sum of the total value of the dissimilarities with respect to each reference pixel and a value representing a variation in the distance between the processing region and each reference pixel is minimized. A reference pixel is determined from the reference pixel candidates for each pixel in the processing region or a predetermined number of pixel groups.

  According to the present invention, it is possible to reduce the amount of code necessary for encoding a prediction residual accompanying improvement in prediction accuracy in the frequency domain by suppressing the occurrence of an unnatural high-frequency component in a predicted image. it can. When a reference pixel is set for each pixel in the prediction target region, a predicted image is generated by collecting pixels that were not originally continuous. As a result, the continuity between pixels in the predicted image is reduced, and a high-frequency component that cannot be a natural image may occur. Therefore, by suppressing the change in the distance to the reference pixel between adjacent pixels in the prediction target region, by setting a pixel in which the same subject is photographed as a reference pixel as much as possible, an unnatural high-frequency component in the predicted image Can be prevented from occurring. In general encoding, the prediction residual is encoded in the frequency domain, and thus, it is possible to prevent generation of useless high frequency components in the prediction residual and realize more efficient encoding. Is possible.

  In the present invention, the reference pixel determining step determines a plurality of reference pixels from the reference pixel candidates for each pixel in the processing region or a predetermined number of pixel groups based on the degree of difference, The predicted image generation step generates a predicted image using a plurality of the reference pixels according to the prediction method.

  According to the present invention, robustness against noise in a reference pixel or a depth map can be improved. If the depth map includes noise, an error may occur when one reference pixel is selected from a plurality of reference pixel candidates. In that case, by selecting a plurality of reference pixels and generating a predicted image using the average value or median of the pixel values for the reference pixel group, several reference pixels in which different subjects are captured by mistake are selected. Even if it is done, the influence can be reduced and a prediction image can be generated. Further, even when noise is included in the reference pixel, it is possible to reduce the noise component by taking an average value using a plurality of reference pixels and to realize more accurate prediction.

  In the present invention, the predicted image generation step sets a weighting factor according to the degree of difference for each of the plurality of reference pixels determined for each pixel in the processing region or for a predetermined number of pixel groups. Including a setting step, and generating a predicted image using a weighted average value or a weighted median value of the plurality of reference pixels using the weighting coefficient according to the prediction method.

  ADVANTAGE OF THE INVENTION According to this invention, the code amount required for the encoding of a prediction residual accompanying the prediction accuracy improvement at the time of performing prediction using a some reference pixel can be reduced. When performing prediction using a plurality of reference pixels, prediction is performed using a weighted average value or a weighted median value using a weighting factor based on the degree of difference between the prediction target pixel and the reference pixel. Therefore, it is possible to improve prediction accuracy while maintaining robustness against noise.

  The present invention decodes a video signal using intra-frame prediction for a processing region obtained by dividing each frame constituting the video into a predetermined size while using a depth map corresponding to the video. In the video decoding method, an intra-frame prediction direction setting step for setting a prediction direction in the intra-frame prediction, and the processing region for each pixel in the processing region or a predetermined number of pixels according to the set prediction direction A reference pixel candidate setting step for setting a pixel within a predetermined distance range from the reference region as a reference pixel candidate, a depth map corresponding to the processing region, and a depth map corresponding to the reference pixel candidate. A reference pixel dissimilarity calculating step for calculating a dissimilarity for each candidate of the reference pixels for each pixel in the region or a predetermined number of pixel groups; A reference pixel determining step for determining a reference pixel from among the reference pixel candidates for each pixel or a predetermined number of pixels in the processing region based on the degree, and from the reference pixel according to the set prediction direction And a predicted image generation step of generating a predicted image for the processing region.

  In the present invention, the reference pixel dissimilarity calculating step includes a difference value between a value of the depth map corresponding to a pixel in the processing region or a predetermined number of pixel groups and a value of the depth map corresponding to the reference pixel candidate. And the distance between the processing region and the reference pixel is used to calculate the degree of difference for each candidate of the reference pixel for each pixel in the processing region or a predetermined number of pixel groups.

  In the present invention, in the reference pixel determining step, the weighted sum of the total value of the dissimilarities with respect to each reference pixel and a value representing a variation in the distance between the processing region and each reference pixel is minimized. A reference pixel is determined from the reference pixel candidates for each pixel in the processing region or a predetermined number of pixel groups.

  In the present invention, the reference pixel determining step determines a plurality of reference pixels from the reference pixel candidates for each pixel in the processing region or a predetermined number of pixel groups based on the degree of difference, The predicted image generation step generates a predicted image using a plurality of the reference pixels according to the prediction direction.

  In the present invention, the predicted image generation step sets a weighting factor according to the degree of difference for each of the plurality of reference pixels determined for each pixel in the processing region or for a predetermined number of pixel groups. Including a setting step, and generating a predicted image using a weighted average value or a weighted median value of the plurality of reference pixels using the weighting coefficient according to the prediction direction.

  The present invention is a video in which predictive encoding is performed using intra-frame prediction on a processing region obtained by dividing each frame constituting the video into a predetermined size while using a depth map corresponding to the video. An encoding apparatus, an intra-frame prediction direction setting means for setting a prediction direction in the intra-frame prediction, and the processing region for each pixel or a predetermined number of pixels in the processing region according to the set prediction direction The reference pixel candidate setting means for setting a pixel within a predetermined distance range from the pixel as a reference pixel candidate, a depth map corresponding to the processing region, and a depth map corresponding to the reference pixel candidate. A reference pixel dissimilarity calculating means for calculating a dissimilarity for each candidate of the reference pixel for each pixel in the region or a predetermined number of pixel groups, and based on the dissimilarity Reference pixel determining means for determining a reference pixel from the reference pixel candidates for each pixel in the processing region or a predetermined number of pixels, and prediction from the reference pixel to the processing region according to the set prediction direction And a predicted image generating means for generating an image.

  In the present invention, the reference pixel dissimilarity calculating means may calculate a difference value between a value of the depth map corresponding to a pixel in the processing region or a predetermined number of pixel groups and a value of the depth map corresponding to the reference pixel candidate. And the distance between the processing region and the reference pixel is used to calculate the degree of difference for each candidate of the reference pixel for each pixel in the processing region or a predetermined number of pixel groups.

  In the present invention, the reference pixel determining means may minimize the weighted sum of the total value of the differences with respect to each reference pixel and a value representing a variation in distance between the processing region and each reference pixel. A reference pixel is determined from the reference pixel candidates for each pixel in the processing region or a predetermined number of pixel groups.

  In the present invention, the reference pixel determination means determines a plurality of reference pixels from the reference pixel candidates for each pixel in the processing region or a predetermined number of pixel groups based on the degree of difference, The predicted image generation means generates a predicted image using a plurality of the reference pixels according to the prediction method.

  In the present invention, the predicted image generation means sets a weighting factor according to the degree of difference for each of the plurality of reference pixels determined for each pixel in the processing region or a predetermined number of pixel groups. Including a setting unit, and generating a predicted image using a weighted average value or a weighted median value of the plurality of reference pixels using the weighting coefficient according to the prediction method.

  The present invention decodes a video signal using intra-frame prediction for a processing region obtained by dividing each frame constituting the video into a predetermined size while using a depth map corresponding to the video. A video decoding apparatus, wherein an intra-frame prediction direction setting unit that sets a prediction direction in the intra-frame prediction, and the processing region for each pixel or a predetermined number of pixels in the processing region according to the set prediction direction The reference pixel candidate setting means for setting a pixel within a predetermined distance range from the pixel as a reference pixel candidate, a depth map corresponding to the processing region, and a depth map corresponding to the reference pixel candidate. A reference pixel dissimilarity calculating means for calculating a dissimilarity for each candidate of the reference pixel for each pixel in the region or a predetermined number of pixel groups, and based on the dissimilarity A reference pixel determining unit that determines a reference pixel from among the reference pixel candidates for each pixel or a predetermined number of pixels in the processing region, and from the reference pixel to the processing region according to the set prediction direction. And a predicted image generating means for generating a predicted image.

  In the present invention, the reference pixel dissimilarity calculating means may calculate a difference value between a value of the depth map corresponding to a pixel in the processing region or a predetermined number of pixel groups and a value of the depth map corresponding to the reference pixel candidate. And the distance between the processing region and the reference pixel is used to calculate the degree of difference for each candidate of the reference pixel for each pixel in the processing region or a predetermined number of pixel groups.

  In the present invention, the reference pixel determining means may minimize the weighted sum of the total value of the differences with respect to each reference pixel and a value representing a variation in distance between the processing region and each reference pixel. A reference pixel is determined from the reference pixel candidates for each pixel in the processing region or a predetermined number of pixel groups.

  In the present invention, the reference pixel determination means determines a plurality of reference pixels from the reference pixel candidates for each pixel in the processing region or a predetermined number of pixel groups based on the degree of difference, The predicted image generation means generates a predicted image using a plurality of the reference pixels according to the prediction direction.

  In the present invention, the predicted image generation means sets a weighting factor according to the degree of difference for each of the plurality of reference pixels determined for each pixel in the processing region or a predetermined number of pixel groups. Including a setting unit, and generating a predicted image using a weighted average value or a weighted median value of the plurality of reference pixels using the weighting coefficient according to the prediction direction.

  The present invention is a video encoding program for causing a computer to execute the video encoding method.

  The present invention is a video decoding program for causing a computer to execute the video decoding method.

  According to the present invention, when a video is transmitted together with data having a value that greatly depends on a subject, such as a depth map for the video, a block including occlusion and noise at the block end, a plurality of blocks with respect to a prediction direction, This makes it possible to perform efficient intra-frame prediction on a block in which a subject exists, and to reduce the amount of codes.

It is a block diagram which shows the structure of the video coding apparatus in one Embodiment of this invention. It is a flowchart which shows the processing operation of the video coding apparatus shown in FIG. It is a block diagram which shows the structure of the video decoding apparatus in one Embodiment of this invention. It is a flowchart which shows the processing operation of the video decoding apparatus shown in FIG. It is a figure which shows the hardware structural example in the case of comprising a video coding apparatus with a computer and a software program. FIG. 25 is a diagram illustrating a hardware configuration example in a case where a video decoding device is configured by a computer and a software program. It is a figure which shows the candidate of a reference pixel at the time of using the vertically downward intra prediction. It is a figure which shows the example at the time of expressing a reference pixel using a reference line. It is a figure which shows the example which set the candidate of the reference pixel with the angle width when using the vertically downward intra prediction. It is a figure which shows the example which made the candidate of a reference pixel with a pixel width when using the vertically downward intra prediction.

  Hereinafter, a video encoding device and a video decoding device according to an embodiment of the present invention will be described with reference to the drawings.

<Video encoding device>
FIG. 1 is a block diagram showing a configuration of a video encoding apparatus in the embodiment. The video encoding device 100 includes an encoding target video input unit 101, an input frame memory 102, a depth map input unit 103, a depth map memory 104, a prediction direction setting unit 105, a reference pixel determination unit 106, a predicted image generation unit 107, a prediction A direction encoding unit 108, a video signal encoding unit 109, a video signal decoding unit 110, a reference frame memory 111, and a multiplexing unit 112 are provided.

  The encoding target video input unit 101 inputs video to be encoded from the outside. In the following description, the video to be encoded is referred to as an encoding target video, and a frame to be processed in particular is referred to as an encoding target frame or an encoding target image. The input frame memory 102 stores the input encoding target video. The depth map input unit 103 inputs a depth map corresponding to the encoding target video. This depth map represents the depth of the subject shown in each pixel of each frame of the encoding target video. The depth map memory 104 stores the input depth map.

  The prediction direction setting unit 105 sets a direction in which intra-frame prediction is performed when generating a predicted image. The reference pixel determination unit 106 determines a reference pixel for generating a predicted image for each pixel in the processing region, using a coded pixel group existing in the set prediction direction and a depth map for the processing region. The predicted image generation unit 107 generates a predicted image based on the determined reference pixel.

  The prediction direction encoding unit 108 encodes information (prediction information) indicating the prediction direction set in the prediction direction setting unit 105. The video signal encoding unit 109 predictively encodes the video signal of the encoding target frame using the generated predicted image. The video signal decoding unit 110 decodes the generated code data using the generated predicted image to generate a decoded frame. The reference frame memory 111 stores the generated decoded frame. The multiplexing unit 112 multiplexes the code data of the prediction information and the code data of the video signal and outputs them to the outside.

  Next, the operation of the video encoding device 100 shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing the operation of the video encoding device 100 shown in FIG. Here, a process of encoding one frame in the video to be encoded will be described. By repeating the processing described for each frame, video encoding can be realized.

  First, the encoding target video input unit 101 inputs an encoding target frame and stores the input encoding target frame in the input frame memory 102. On the other hand, the depth map input unit 103 inputs a depth map for the encoding target frame, and stores the input depth map in the depth map memory 104 (step S101). Note that although the input encoding target frames are sequentially encoded here, the input order and the encoding order do not necessarily match. If the input order and the encoding order are different, the input frame and depth map are stored in the input frame memory 102 and the depth map memory 104 until the next frame to be encoded is input. The stored encoding target frame and depth map may be deleted from each memory after being encoded by the encoding process described below.

  Note that the depth map input in step S101 is a depth map obtained on the decoding side, such as a depth map that has already been encoded. This is to suppress the occurrence of coding noise such as drift by using exactly the same information as the information obtained by the video decoding apparatus. However, when the generation of such encoding noise is allowed, the original one before encoding may be input. Examples of depth maps obtained on the other decoding side include a depth map synthesized using a decoded depth map of another viewpoint and a decoded image group of another viewpoint. There is a depth map estimated by stereo matching or the like.

  When the encoding target frame and the depth map are stored, the encoding target frame is divided into regions of a predetermined size, and the video signal of the encoding target frame is encoded for each divided region (step S102). ~ S111). That is, assuming that the encoding target area index is blk and the total number of encoding target areas in one frame is represented by numBlks, blk is initialized to 0 (step S102), and then 1 is added to blk (step S110). ), The following processing (step S103 to step S109) is repeated until blk becomes numBlks (step S111). In general encoding, it is divided into processing unit blocks called macroblocks of 16 pixels × 16 pixels, but may be divided into blocks of other sizes as long as they are the same as those on the decoding side.

  In the process repeated for each encoding target region, the prediction direction setting unit 105 sets a prediction direction used for intra-frame prediction (step S103). The prediction direction is a direction in which already encoded pixels exist that are referred to when predicting the video signal of the encoding target region blk. For each pixel in the encoding target region blk, a prediction image is generated using the decoded pixel value of the already encoded pixel group existing in the source indicated by the prediction direction. The prediction direction set here may be determined in any way. However, when maximizing the coding efficiency, it is better to evaluate the prediction efficiency based on the prediction image generated according to the set prediction direction and determine the one that maximizes the prediction efficiency.

That is, when the prediction direction is v, the prediction direction V _blk given by the equation (1) is determined as the prediction direction for the encoding target region blk. Note that the prediction direction may be expressed in any way. Here, a settable prediction direction candidate is NumV, and the prediction direction is indicated by an index value represented by an integer from 0 to NumV-1. Each index value corresponds to a different direction (two-dimensional vector).

Note that the smaller the value of E _blk (v) is, the higher the encoding efficiency is, and argmin is a function that returns a parameter that minimizes a given function. The parameter to be minimized is given at the bottom of argmin. Any prediction direction candidates may be used. For example, H.M. Similarly to intra prediction in H.264 / AVC 4 × 4 block, eight kinds of prediction directions may be used as candidates, and the document “K. McCann, W.-J. Han, and I. Kim,“ Samsung's Response to the Call ”. for Proposals on Video Compression Technology ", Input document to Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11, JCTVC-A124, April 2010. '' A large number of prediction directions may be used as candidates.

Further, it is not always necessary to be fixed, and the candidate may be changed according to the picture type, block size, or the like. For example, H.M. In H.264 / AVC, 8 types of prediction directions can be used in the 4 × 4 block as described above, but only 3 types of prediction directions can be used in the 16 × 16 block. Any evaluation efficiency evaluation value E _blk (v) may be used. For example, the sum of absolute differences (SAD) or sum of squares of differences (SSD) between the encoding target image and the predicted image represented by the equations (2) and (3) may be used.

Here, Org indicates an encoding target frame, and Pred _V indicates a predicted image generated by a method described later when the prediction direction is v. In addition to these methods, there is a method using a value obtained by converting a difference value between an encoding target image and a predicted image using DCT, Hadamard transform, or the like. If the transformation is represented by matrix A, it can be represented by equation (4). Note that ‖X‖ represents the norm of X.

In addition, as described above, instead of a method for evaluating only the degree of divergence between the encoding target frame and the predicted image, an RD cost in consideration of the generated code amount and distortion amount may be used. The RD cost used here can be expressed by Equation (5) using the code amount R _blk (v) and the distortion amount D _blk (v) when Org is encoded using Pred _V as a predicted image. Note that λ is a Lagrange multiplier, and a predetermined value is used.

In order to calculate an accurate RD cost, it is necessary to encode and decode a prediction residual. Since this requires a large number of operations, the calculation is performed using the code amount R _blk ′ (v) of the prediction information and the distortion amount D _blk ′ (v) from the encoding target frame in the prediction image. The prediction direction v may be determined using the RD cost. At this time, Lagrange's undetermined multiplier often uses a value different from λ. Further, the code amount R _blk ′ is not actually obtained by encoding, but an estimated amount of the code amount R _blk ′ is obtained by a simple method, and it is possible to further increase the speed by using it.

  When the prediction direction is set, the reference pixel determining unit 106 uses the depth map for the encoding target region blk and the reference pixel candidate to generate a reference pixel for generating a prediction image for each pixel in the encoding target region. Is determined (step S104). The reference pixel candidates are set for each pixel in the encoding target area, and are a set of already encoded pixels that are found by tracing the pixels in the encoding target area in the original direction of the prediction direction.

That is, in the case of vertical prediction in which prediction is performed from the upper block, the direction from which the prediction direction is based is up, and as shown in FIG. 7, the already encoded pixels found by tracing up from the encoding target pixel It is a set. In determining the reference pixel, the encoding target region or the distance from the pixel may be used as one of selection criteria. In the present embodiment, a candidate set of reference pixels of a pixel p having a prediction direction v is represented by {rp _{p, v, i} | i = 1, 2,... N _{p, v} }. N _p is the number of reference pixel candidates, and the smaller the value of i, the closer the pixel is to the encoding target pixel.

  Note that the number of reference pixel candidates may be fixed or variable. For example, the number of candidates may be specified in advance, the retroactive distance may be set in advance, or a combination thereof may be specified. The specified condition and number are determined in units of block, slice, frame, GOP (Group Of Pictures), etc., and the value may be encoded at positions such as block header, slice header, picture header, GOP header. Absent. The retroactive distance may be defined as the distance from the end of the encoding target block, or may be defined as the distance from the encoding target pixel. If a fixed condition is always used, it is not necessary to encode the information.

  Further, the reference pixel may be determined by determining refline_x and refline_y shown in FIG. 8 for each pixel in the encoding target region. In the figure, Mx and My are values defined by the maximum distance from the encoding target region or the encoding target pixel to the reference pixel. In this case, a combination of refline_x and refline_y is a reference pixel candidate. However, depending on the prediction direction, only one of refline_x and refline_y may be determined. For example, refline_x is unnecessary when prediction is performed vertically downward, and refline_y is not required when prediction is performed horizontally right.

As represented by Expression (6), the reference pixel is determined as a pixel having a minimum difference from the pixel in the encoding target region among the set reference pixel candidates.

Dissimilarity indicates the degree of dissimilarity between two given pixels. For example, it is possible to use a difference absolute value or a difference square value of the depth map values expressed by the equations (7) and (8). .

It is also possible to use a scale that considers not only the difference in depth value, but also the distance from the encoding target area or the pixels in the encoding target area to the reference pixel. For example, equation (9) may be used.

α is a weighting factor determined separately, and distance indicates the distance between two given pixels. When the above definition is used, if the reference pixel candidate is rp _{p, v, i} , i can be used as the distance. Also, the total value of refline_x and refline_y can be used as the distance. By considering the distance, when the degree of difference due to the depth map value is the same, a pixel closer to the encoding target region is used as a reference pixel. In general, it is considered that the closer the spatial distance is, the higher the correlation is. Therefore, it is possible to select a more appropriate reference pixel by using a scale considering such a distance.

Furthermore, when considering the distance from the encoding target area or the pixel in the encoding target area to the reference pixel, the change of the distance for each pixel in the encoding target area may be considered. In this case, the reference pixel for each pixel in the encoding target area is given by the expression (1 0 ).

Here, the set X is a set of reference pixels for the pixels in the encoding target region, and x _p represents the _{p-th element} of X, that is, the reference pixels for the pixel p. The reference pixel for the pixel p is selected from the candidates {rp _{p, v, i} | i = 1, 2,... N _{p, v} } as described above. E is a set representing the adjacency relationship between pixels in the encoding target region, and it is considered that the pixel p and the pixel q are adjacent when (p, q) εE. Further, _hpq is a smoothing term for suppressing the variation of the reference pixels selected for each pixel, and any one can be used here as long as it is an increasing function with respect to the distance between the reference pixels. Note that the distance between the reference pixels may be calculated and used, or the distance between the reference pixel and the encoding target pixel for each reference pixel may be obtained and the difference between them may be used. . The difference in distance between the reference pixel and the encoding target pixel with respect to each reference pixel is that when x _p is rp _{p, v, i} and x _q is rp _{q, v, j} , the distance between the reference pixels is | i −j |.

  The minimization of equation (10) can be solved using an algorithm that solves the Markov random field minimization problem such as graph cut. However, since it may be difficult to find a solution that minimizes the right side in a strict sense, an approximate solution may be used. When an approximate solution is used, it is necessary to obtain a similar solution on the decoding side. Therefore, it is necessary to stop minimization using a predetermined condition or to encode and transmit a stop condition for minimization. is there.

  In addition, in equation (10), minimization is performed for all pixels in the encoding target area, but in consideration of the amount of calculation, for each subset such as for each row or column in the encoding target area. It is also possible to perform minimization and use the result as an approximate solution. In particular, when the adjacency relationship is limited to one dimension, the minimum solution for the subset can be found at high speed by using dynamic programming.

Next, when the reference pixel is determined, the predicted image generation unit 107 generates a predicted image (step S105). A prediction image is generated by assigning a pixel value of a reference pixel for each pixel, as in general intra-frame prediction. That is, the predicted image is generated based on the equation (11). Dec indicates a decoded image corresponding to an already encoded area of the encoding target image stored in the reference frame memory 111.

  Next, when the generation of the prediction image is completed, the prediction direction encoding unit 108 performs encoding of the prediction direction (step S106). Any method may be used for encoding the prediction direction. However, in order to enable correct decoding, it is necessary to use a method that can be decoded on the decoding side. For example, the index value in the prediction direction may be binarized using a predetermined method and encoded by binary arithmetic coding. In addition, a prediction value of the prediction direction is generated from the prediction direction of the intra-frame prediction used when coding the adjacent block, and in addition to whether the prediction is correct or not, the prediction direction only when the prediction is incorrect There is also a method of encoding.

  Next, using the generated predicted image, the video signal encoding unit 109 encodes the video signal in the encoding target region blk of the encoding target frame Org (step S107). Any method may be used for encoding as long as decoding is possible on the decoding side. MPEG-2 and H.264 In general encoding such as H.264 / AVC, encoding is performed by sequentially performing frequency conversion such as DCT, quantization, binarization, and entropy encoding on a difference signal between a video signal of a block blk and a predicted image. Do.

  Next, the video signal decoding unit 110 decodes the video signal for the block blk using the code data of the video signal and the predicted image, and the decoded frame Dec [blk] as a decoding result is stored in the reference frame memory 111. Store (step S108). Here, a method corresponding to the method used at the time of encoding is used. For example, MPEG-2 and H.264. In general encoding such as H.264 / AVC, the code data is subjected to frequency inverse transform such as entropy decoding, inverse binarization, inverse quantization, and IDCT in order, and the obtained two-dimensional signal Then, the predicted image is added, and finally the video signal is decoded by performing clipping in the pixel value range.

  Note that the data immediately before the process on the encoding side becomes lossless and the predicted image may be received, and the decoding process may be performed by a simplified decoding process. That is, in the above-described example, a two-dimensional signal obtained by receiving a value obtained by applying quantization processing at the time of encoding and a predicted image, and sequentially performing inverse quantization and frequency inverse transform on the quantized value. It is also possible to decode a video signal by adding a predicted image to the image and clipping in the range of pixel values.

  Next, the multiplexing unit 112 multiplexes and outputs the code data in the prediction direction and the code data of the video signal (step S109). In addition, although it multiplexed for every block here, you may multiplex by a frame unit. However, in that case, it is necessary to decode the code data for one frame at the time of decoding.

  Note that the order of some processes shown in FIG. 2 may be changed. Specifically, the encoding process of the prediction direction in step S106 may be performed any time after the determination process of the prediction direction in step S103, and the multiplexing process of the code data in step S109 is performed in step S106. Any timing may be used as long as it is after the encoding process in the prediction direction and the encoding process of the video signal in step S107.

<Video decoding device>
Next, the video decoding device will be described. FIG. 3 is a block diagram showing a configuration of a video decoding apparatus according to an embodiment of the present invention. The video decoding apparatus 200 includes a code data input unit 201, a code data memory 202, a depth map input unit 203, a depth map memory 204, a separation unit 205, a prediction direction decoding unit 206, a reference pixel determination unit 207, a predicted image generation unit 208, A video signal decoding unit 209 and a reference frame memory 210 are provided.

  The code data input unit 201 inputs code data of a video to be decoded from the outside (for example, encoded data output from the video encoding device 100 illustrated in FIG. 1). In the following description, the video to be decoded is referred to as a decoding target video, and a frame to be processed in particular is referred to as a decoding target frame or a decoding target image. The code data memory 202 stores the input code data. The depth map input unit 203 inputs a depth map corresponding to the decoding target video. This depth map represents the depth of the subject shown in each pixel of each frame of the decoding target video. The depth map memory 204 stores the input depth map.

  The separation unit 205 separates the code data of the prediction information multiplexed with the input code data and the code data of the video signal. The prediction direction decoding unit 206 decodes information indicating the direction (prediction direction) in which intra-frame prediction is performed when generating a predicted image from the code data. The reference pixel determination unit 207 determines, for each pixel in the processing region, a reference pixel for generating a predicted image, using the decoded pixel group existing in the decoded prediction direction and the depth map for the processing region. The predicted image generation unit 208 generates a predicted image based on the determined reference pixel. The video signal decoding unit 209 decodes the code data using the generated predicted image to generate a decoded frame. The reference frame memory 210 stores the generated decoded frame.

  Next, the operation of the video decoding apparatus 200 shown in FIG. 2 will be described with reference to FIG. FIG. 4 is a flowchart showing the operation of the video decoding apparatus 200 shown in FIG. Here, a process of decoding one frame in the decoding target video will be described. By repeating the processing described for each frame, video decoding can be realized.

  First, the code data input unit 201 inputs code data of a decoding target video, and stores the input code data in the code data memory 202. On the other hand, the depth map input unit 203 inputs a depth map for the decoding target video, and stores the input depth map in the depth map memory 204 (step S201). Although the decoding target frames are sequentially decoded and output from the input code data here, the input order and the output order are not necessarily the same. If the input order and the output order are different, the decoded frame is stored in the reference frame memory 210 until the next output frame is decoded. The stored depth map may be deleted from each memory when the corresponding frame is decoded by the decoding process described below.

  Note that the depth map input in step S201 is the same as the depth map used during encoding. This is to suppress the occurrence of encoding noise such as drift by using exactly the same information as that used in the video encoding apparatus 100. However, when the generation of such encoding noise is allowed, a different one from that used for encoding may be input. As the input depth map, for example, a depth map decoded separately, a depth map synthesized using a depth map decoded for another viewpoint, or an image group decoded for another viewpoint There is a depth map estimated by stereo matching or the like.

  Next, when the code data and the depth map are stored, the video signal of the decoding target frame is decoded for each area obtained by dividing the decoding target frame into a predetermined size (steps S202 to S210). That is, assuming that the decoding target region index is blk, and the total number of decoding target regions in one frame is represented by numBlks, blk is initialized with 0 (step S202), and then 1 is added to blk (step S209). The following processing (step S203 to step S208) is repeated until blk becomes numBlks (step S210). The size of the processing area is the same as that used on the encoding side. In general encoding, a processing unit block called a 16 × 16 pixel macroblock is used, but if it is the same as the encoding side, processing is performed for each block of other sizes.

  In the process repeated for each decoding target area, first, the separation unit 205 separates the code data in the prediction direction of the decoding target area blk and the code data of the video signal from the code data (step S203). Note that although the decoding target areas are separated here, they may be separated in other units such as a frame unit. However, when separation is performed in units of frames, it is necessary to store the separated code data, not the input code data.

  Next, the prediction direction decoding unit 206 decodes the encoded data for the separated prediction direction, and sets the prediction direction of intra-frame prediction for the decoding target region blk (step S204). After decoding in the prediction direction, the reference pixel determining unit 207 determines a reference pixel for generating a predicted image for each pixel in the decoding target area, using a depth map for the decoding target area blk and the reference pixel candidate. (Step S205). The process here is the same as step S104 shown in FIG. The reference pixel may be determined according to any rule, but it is necessary to determine according to the same rule as that on the encoding side. If information about the rules used on the encoding side is encoded, the information is decoded at an appropriate timing (the head of the sequence, frame, block group, block, etc.) and referenced according to the decoded rules. It is necessary to determine the pixel.

  Next, when the determination of the reference pixel for each pixel in the decoding target area is completed, the predicted image generation unit 208 generates a predicted image (step S206). The process here is the same as step S105 shown in FIG. 2, and a predicted image is generated by assigning a pixel value of a reference pixel for each pixel, as in general intra-frame prediction.

  Next, when the generation of the predicted image is completed, using the generated predicted image, the video signal decoding unit 209 decodes the video signal for the decoding target area blk from the code data for the video signal (step S207), Is generated (step S208). The decoded image becomes an output of the video decoding device 200 and is stored in the reference frame memory 210. The decoding process uses a method corresponding to the method used at the time of encoding. For example, MPEG-2 and H.264. When general coding such as H.264 / AVC is used, the code data is subjected to frequency inverse transformation such as entropy decoding, inverse binarization, inverse quantization, and IDCT in order, and the obtained 2 The predicted image is added to the dimension signal, and finally, the video signal is decoded by performing clipping in the pixel value range.

In the above description, one prediction direction is set for the entire encoding target region and the entire decoding target region (hereinafter, encoding and decoding are not distinguished from each other), but the processing target region is divided. However, reference information may be set for each division. In this case, the method for dividing the processing target region is also a part of information for indicating the prediction direction. That is, the information indicating the prediction direction includes information P _blk indicating the division method and a set of prediction directions for each division {V _{blk, i} | i = 1,..., M _blk }. M _blk represents the number of divisions when the processing target area is divided by the division method P _blk . In this way, even when the subject captured in each part is different within the processing target area, that is, when the direction of the spatial correlation of the video signal is greatly different, the prediction image is generated in consideration thereof. Can do. Even in this case, by determining the reference pixel using the depth map, it is possible to generate a predicted image reflecting the subject reflected in each pixel without setting a fine region division.

In addition, the reference pixel is determined for each pixel in the processing region. When the reference pixel is determined by determining the distance from the encoding target region, that is, the reference line (refline_x and refline_y), for each pixel in the processing region Instead, the reference pixel may be determined for each pixel group such as a row (horizontal line) or column (vertical line) in the processing region, or an N × N pixel block. That is, one reference line (or a set of reference lines, hereinafter referred to as a reference line without distinguishing both) may be determined for each pixel group determined in advance or separately. In this case, the process of determining the reference pixel in step S104 and step S205 is performed according to equation (12) instead of equation (6) described above.

  Here, subblk represents a set of target pixel groups for determining one reference line. refline represents either refline_x or refline_y when determining one of the processing line left or reference line on the processing area, and refline_x or refline_y when determining a reference line pair. Represents. Further, refp (p, v, rl) indicates a reference pixel when performing intra-frame prediction with respect to the pixel p in the prediction direction v and the reference line rl. When p1 ≠ p2, refp (p1, v, rl) = refp (p2, v, rl) is not always satisfied. If the positional relationship between p1 and p2 is parallel to the prediction direction v, refp (p1, v, rl) = refp (p2, v, rl). This is because the reference line is defined by the distance from the processing coping area.

In addition, by setting a single reference line for each defined pixel group in consideration of changes in the reference line in the processing region, as in the reference pixel determination process using the above-described equation (10), It is also possible to determine the reference pixel. The mathematical formula in that case is given by equation (13).

Here, the set RL is a set of reference lines, and rl _sb indicates a reference line for the pixel set sb of the unit for setting the reference line, that is, the _{sb element} of RL. E ′ is a set representing the adjacency relationship between the pixel sets of the unit for setting the reference line in the processing target area. When (s, t) ∈E ′, the pixel set s and the pixel set t are considered to be adjacent. . Further, h _st is a smoothing term for suppressing the variation of the reference line determined for each pixel set. Here, any function may be used as long as it is an increasing function with respect to the distance between the reference lines. For example, a value obtained by scaling the sum of absolute differences of reference lines with a predetermined coefficient may be used. The sum of absolute differences of reference lines is the sum of absolute values of differences between the values of refline_x and refline_y in each reference line.

  In the above description, only intra-frame prediction performed by specifying a prediction direction is targeted. However, within a frame in which a prediction value is generated from a processed pixel group adjacent to a processing target region without specifying a prediction direction. It can also be applied to prediction. Specifically, the present invention can be applied by considering the process of determining the reference pixels as the process of determining the reference lines (refline_x and refline_y). In this case, the process of determining the reference pixels in step S104 and step S205 may use the equation (12) or determine the reference line with subblk = blk.

As another method, the difference between the representative value of the depth map for the processing target region and the value of the depth map for each pixel of the reference pixel group used to generate the predicted value specified by the reference line. The reference line may be determined so that the total value of the degrees is minimized. The mathematical formula in that case is given by formula (14).

  Here, repD indicates a representative value of the depth map value for the processing target region, and an average value, a median value, a most abundant value, or the like of the depth map value for the region can be used. ref (rl) indicates a reference pixel group when the reference line is rl. Although the difference absolute value is used as the divergence degree, an arbitrary function can be used as long as it is an increase function for the difference in the depth map value such as the difference square value. Also, instead of calculating the degree of divergence for each reference pixel and evaluating the total value, the representative value of the depth map value for the reference pixel group is obtained, and the representative value and the representative value of the depth map value for the processing target area are obtained. There is also a method of evaluating by the degree of deviation from the value.

  In the above description, as illustrated in FIG. 7, a set of already processed pixels that are found by tracing in the direction of the prediction direction with respect to the pixels in the processing region are used as reference pixel candidates. However, by using a prediction direction having a certain angular width as shown in FIG. 9, it is possible to make more pixels as reference pixel candidates. In addition, there is a method in which, as shown in FIG. 10, instead of the angular width, the already processed pixels found by tracing in the direction of the prediction direction and the surrounding pixel groups are used as reference pixel candidates. Thus, by increasing the number of reference pixel candidates in the direction orthogonal to the prediction direction, it is possible to set the reference pixels used for prediction in a wider range, thereby improving the prediction performance. In addition, since a large number of directions can be covered in one direction, it is possible to reduce the number of prediction directions that can be set and to reduce the amount of code necessary to encode information indicating the prediction direction. .

  In the above description, one of the plurality of reference pixel candidates is selected, but a plurality of reference pixels may be set. As a method of setting a plurality of reference pixels, a method of setting a certain number of reference pixels as a reference pixel in order from the smallest difference based on the degree of difference from the pixel of the encoding target region among the reference pixel candidates, There are a method of setting a reference pixel that has a difference degree equal to or less than a certain value, and a method of setting a certain number of pixels as a reference pixel in order from the smallest difference value that is less than a certain value. In addition, as a method for generating a pixel value (predicted value) of a predicted image from a plurality of reference pixels, a method of using an average value or median value of a plurality of reference pixels as a predicted value, or a weighting factor for each reference pixel depending on the degree of difference There is a method in which a weighted average value or a weighted median value in which is set as a predicted value. Although any method including the above can be used as a method for setting a plurality of reference pixels and a method for generating a prediction value from a plurality of reference pixels, it is necessary to match between encoding and decoding.

  In the above description, the input depth map is used as it is. However, when an encoded depth map is used, noise is generated in the depth map. Therefore, the depth map is used to reduce the noise. You may apply a low-pass filter to. In addition, if there is a bit depth that can detect the difference between subjects, a predicted image can be generated in consideration of the subject. Therefore, bit depth conversion is performed on the input depth map to reduce the bit depth of the depth map. Processing may be performed. Note that simple bit depth conversion may be performed, but the number of subjects may be determined from the depth map, and the information may be converted into information that only distinguishes the subjects according to the result.

  In the above description, the entire frame is encoded / decoded. However, the present invention can be applied to only a part of an image. In this case, it may be determined whether or not to apply the process, and a flag indicating the process may be encoded / decoded, or may be designated by some other means.

  In addition, instead of designating information separately, it may be determined whether or not to determine the reference pixel described above from the depth map. For example, when the depth map value for each pixel in the processing target area is substantially equal (when the variance value of the depth map value is small), only one subject is included, and therefore always adjacent to the processing target area. There is a method in which the reference pixel is not determined using the value of the depth map, assuming that the processed pixel is the reference pixel. In addition to the pixels in the processing target region, a pixel group set as a reference pixel is included when performing prediction using a conventional method (such as H.264 / AVC) according to the set prediction direction. It may be determined whether or not the corresponding depth map values are approximately equal for the pixel set. In this way, it is possible to use more accurate switching because it is possible to use whether or not only one subject is included in the area including the reference pixel group.

  In the above description, the depth map for the video to be encoded / decoded is used. However, image information having a value depending on the subject such as a normal map or a temperature image can be used instead. However, what is used on the encoding side needs to be available on the decoding side as well.

  The video encoding and video decoding processes described above can also be realized by a computer and a software program. The program can be provided by being recorded on a computer-readable recording medium or provided via a network. Is possible.

  FIG. 5 shows a hardware configuration example in the case where the image encoding apparatus is configured by a computer and a software program. This system includes a CPU 50 that executes a program, a memory 51 such as a RAM that stores programs and data accessed by the CPU 50, and an encoding target video input unit 52 that inputs an encoding target video signal from a camera or the like. A storage unit that stores a video signal from a disk device or the like) and a depth map input unit 53 that inputs a depth map for a video to be encoded via a network, for example (a storage unit that stores a depth map from a disk device or the like). ), And a program storage device 54 storing a video encoding program 541 that is a software program for causing the CPU 50 to execute the processing shown in FIG. 2, and the CPU 50 executing the video encoding program 541 loaded in the memory 51. Generate code data, for example, network A code data output unit 55 to be output (or a storage unit for storing the multiplexed code data by the disc unit, etc.), have become connected to each other by a bus.

  Although not shown, other hardware such as a code data storage unit and a reference frame storage unit is provided and used to implement this method. Also, a video signal code data storage unit, a prediction direction code data storage unit, and the like may be used.

FIG. 6 shows an example of a hardware configuration when the video decoding apparatus is configured by a computer and a software program. This system includes a CPU 60 that executes a program, a memory 61 such as a RAM that stores programs and data accessed by the CPU 60, and a code data input unit 62 that inputs code data encoded by the video encoding apparatus according to the present method. (It may be a storage unit that stores multiplexed code data by a disk device or the like) and, for example, a depth map input unit 63 that inputs a depth map for a video to be decoded via a network (a storage that stores a depth map by a disk device or the like) A program storage device 64 storing a video decoding program 641 which is a software program for causing the CPU 60 to execute the processing shown in FIG. 4, and the CPU 60 executing the video decoding program 641 loaded in the memory 61. Obtained by decoding the code data The issue picture, the decoded video output unit 65 to output to the reproduction unit has the connecting configurations bus.

  Although not shown in the drawing, other hardware such as a reference frame storage unit is provided and used to implement this method. Also, a video signal code data storage unit and a prediction direction code data storage unit may be used.

  As described above, according to the video encoding device and the video decoding device described above, it is possible to reduce the amount of code necessary for encoding the prediction residual accompanying the improvement of the prediction accuracy of the video signal. Occlusion at the edge of the region is enabled not only by the pixels adjacent to the prediction target region in intraframe prediction, but also by prediction from pixels located within a certain distance and along the prediction direction of intraframe prediction. In addition, pixel values from the same subject can be predicted even in a case where noise is included or in a region where there are a plurality of subjects in the prediction direction, so that highly accurate prediction can be realized. In the conventional method (for example, Patent Document 1), the distance from the prediction target region is specified, and intra-frame prediction using pixels other than the pixels adjacent to the prediction target region is realized. In this case, since it is necessary to encode information for specifying the distance, the distance cannot be specified for each pixel, and high-precision prediction cannot be performed for all the pixels. In the present invention, using the depth map information transmitted separately, a pixel in which the same subject as the pixel to be predicted is captured is selected from the reference pixel candidates, and the selected reference pixel is used. To generate a predicted image. In this way, it is not necessary to encode information specifying the distance. As a result, it is possible to determine the distance for each pixel, and it is possible to realize highly accurate prediction for all the pixels. Here, since the value of the depth map is largely dependent on the subject, it is possible to determine whether or not the same subject is captured with high accuracy by determining the subject using the depth map. Even when the distance is designated for each pixel group, it is not necessary to encode the information for designating the distance, which is necessary in Patent Document 1, so that the amount of code can be reduced.

  In addition, it is possible to reduce the amount of code necessary for encoding the prediction residual accompanying improvement in prediction accuracy by performing prediction from a pixel having a higher correlation with respect to the pixel to be predicted. Since the texture information is lost in the depth map, it is not possible to determine the reference pixel taking into account the texture in the same subject. In general, the closer the spatial distance in the image, the higher the correlation between the pixel values. Therefore, in addition to the depth map value, the reference pixel is determined based on the distance standard between the reference pixel and the prediction target pixel. When there are a plurality of pixels in which the same subject appears in the reference image candidates, prediction using pixels with higher correlation becomes possible.

  In addition, it is possible to reduce the amount of code necessary for encoding the prediction residual accompanying improvement in prediction accuracy in the frequency domain by suppressing the occurrence of an unnatural high-frequency component in the predicted image. When a reference pixel is set for each pixel in the prediction target region, a predicted image is generated by collecting pixels that were not originally continuous. As a result, the continuity between pixels in the predicted image is reduced, and a high-frequency component that cannot be a natural image may occur. Therefore, by suppressing the change in the distance to the reference pixel between adjacent pixels in the prediction target region, by setting a pixel in which the same subject is photographed as a reference pixel as much as possible, an unnatural high-frequency component in the predicted image Can be prevented from occurring. In general encoding, the prediction residual is encoded in the frequency domain, and thus, it is possible to prevent generation of useless high frequency components in the prediction residual and realize more efficient encoding. Is possible.

  In addition, robustness against noise in the reference pixel and the depth map can be improved. If the depth map includes noise, an error may occur when one reference pixel is selected from a plurality of reference pixel candidates. In that case, by selecting a plurality of reference pixels and generating a predicted image using the average value or median of the pixel values for the reference pixel group, several reference pixels in which different subjects are captured by mistake are selected. Even if it is done, the influence can be reduced and a prediction image can be generated. Further, even when noise is included in the reference pixel, it is possible to reduce the noise component by taking an average value using a plurality of reference pixels and to realize more accurate prediction.

  In addition, it is possible to reduce the amount of code necessary for encoding the prediction residual accompanying improvement in prediction accuracy when performing prediction using a plurality of reference pixels. When performing prediction using a plurality of reference pixels, prediction is performed using a weighted average value or a weighted median value using a weighting factor based on the degree of difference between the prediction target pixel and the reference pixel. Therefore, it is possible to improve prediction accuracy while maintaining robustness against noise.

  A program for realizing the functions of the processing units in FIGS. 1 and 3 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed to execute an image. Encoding processing and video decoding processing may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Accordingly, additions, omissions, substitutions, and other modifications of components may be made without departing from the spirit and scope of the present invention.

  In encoding free-viewpoint video data having video and depth maps as components, it can be applied to applications where it is essential to improve the encoding efficiency of intra-frame prediction.

  DESCRIPTION OF SYMBOLS 100 ... Video coding apparatus ..., 101 ... Encoding target video input unit, 102 ... Input frame memory, 103 ... Depth map input unit, 104 ... Depth map memory, 105 ..Prediction direction setting unit 106. Reference pixel determination unit 107 107 Prediction image generation unit 108. Prediction direction encoding unit 109 109 Video signal encoding unit 110 110 Video Signal decoding unit, 111 ... reference frame memory, 112 ... multiplexing unit, 200 ... video decoding device, 201 ... code data input unit, 202 ... code data memory, 203 ... depth Map input unit, 204 ... Depth map memory, 205 ... Separation unit, 206 ... Prediction direction decoding unit, 207 ... Reference pixel determination unit, 208 ... Prediction image generation unit, 209 ... Video signal recovery Department, 210 ... reference frame memory

Claims (22)

A video encoding method that performs predictive encoding using intra-frame prediction on a processing region obtained by dividing each frame constituting the video into a predetermined size while using a depth map corresponding to the video. There,
An intra-frame prediction direction setting step for setting a prediction direction in the intra-frame prediction;
For each pixel in the processing area or for a predetermined number of pixels, a pixel that exists in a direction opposite to the set prediction direction and is already encoded within a predetermined distance range from the processing area is a reference pixel candidate. A reference pixel candidate setting step to be set as
Using the depth map corresponding to the processing region and the depth map corresponding to the reference pixel candidate, a difference degree for each reference pixel candidate is calculated for each pixel in the processing region or a predetermined number of pixel groups. A reference pixel difference calculating step,
A reference pixel determination step of determining a reference pixel from among the reference pixel candidates for each pixel in the processing region or a predetermined number of pixel groups based on the degree of difference;
A prediction image generation step of generating a prediction image for the processing region from the reference pixel according to the set prediction direction. In the reference pixel candidate setting step, in addition to the reference pixel candidate, a direction orthogonal to the set prediction direction with respect to the reference pixel candidate for each pixel in the processing region or a predetermined number of pixel groups. 2. The video encoding method according to claim 1, wherein an already encoded pixel that is present in a predetermined distance range from the reference pixel candidate is also set as the reference pixel candidate. In the reference pixel candidate setting step, in addition to the reference pixel candidates, a predetermined angle range centered on a direction opposite to the set prediction direction is set for each pixel in the processing region or for a predetermined number of pixel groups. 2. The video encoding method according to claim 1, wherein an already encoded pixel that exists and is within a predetermined distance range from the processing region is also set as the reference pixel candidate. The reference pixel dissimilarity calculating step includes a difference value between a depth map value corresponding to a pixel in the processing region or a predetermined number of pixel groups and a depth map value corresponding to the reference pixel candidate, and the processing by using the distance between the reference pixel and area, for each pixel or a predetermined number of pixel groups of the processing region, from claim 1, characterized in that to calculate the dissimilarity for each candidate of the reference pixels 3 The video encoding method according to any one of the above. The reference pixel determination step, the sum is minimized with increasing function value for the distance between the reference pixel for smoothing the sum of the dissimilarity, the variation of the reference picture element in the processing region for each reference pixel as, for each pixel or a predetermined number of pixel groups of said processing region, the image of any one of claims 1 to 4, characterized in that to determine the reference pixels from the candidates of the reference pixel Encoding method. The reference pixel determining step determines a plurality of reference pixels from the reference pixel candidates for each pixel in the processing region or a predetermined number of pixel groups based on the degree of difference,
The predicted image generating step, the following prediction Direction, video encoding method according to any one of claims 1 5, characterized in that to generate a predictive image using a plurality of reference pixels. The predicted image generation step includes a weighting factor setting step of setting a weighting factor according to the degree of difference for each of the plurality of reference pixels determined for each pixel in the processing region or for a predetermined number of pixel groups. ,
In accordance with the prediction Direction, video encoding method according to claim 6, characterized in that generating the predicted image by using a weighted average or weighted median of the plurality of reference pixels using the weighting factor . A video decoding method that decodes a video signal using intra-frame prediction on a processing area obtained by dividing each frame constituting the video into a predetermined size while using a depth map corresponding to the video. There,
An intra-frame prediction direction setting step for setting a prediction direction in the intra-frame prediction;
For each pixel or a predetermined number of pixel groups of said processing region, present in the prediction direction opposite the direction that the setting, as already candidates for reference pixels decoded pixels within a predetermined distance range from the processing region A reference pixel candidate setting step to be set;
Using the depth map corresponding to the processing region and the depth map corresponding to the reference pixel candidate, a difference degree for each reference pixel candidate is calculated for each pixel in the processing region or a predetermined number of pixel groups. A reference pixel difference calculating step,
A reference pixel determination step of determining a reference pixel from among the reference pixel candidates for each pixel in the processing region or a predetermined number of pixel groups based on the degree of difference;
A predictive image generating step of generating a predictive image for the processing region from the reference pixel according to the set prediction direction. In the reference pixel candidate setting step, in addition to the reference pixel candidate, a direction orthogonal to the set prediction direction with respect to the reference pixel candidate for each pixel in the processing region or a predetermined number of pixel groups. The video decoding method according to claim 8, wherein pixels that have already been decoded and are within a predetermined distance range from the reference pixel candidate are also set as the reference pixel candidates. In the reference pixel candidate setting step, in addition to the reference pixel candidates, a predetermined angle range centered on a direction opposite to the set prediction direction is set for each pixel in the processing region or for a predetermined number of pixel groups. 9. The video decoding method according to claim 8, wherein pixels that exist and have already been decoded within a predetermined distance range from the processing area are also set as candidates for the reference pixel. The reference pixel dissimilarity calculating step includes a difference value between a depth map value corresponding to a pixel in the processing region or a predetermined number of pixel groups and a depth map value corresponding to the reference pixel candidate, and the processing by using the distance between the reference pixel and area, for each pixel or a predetermined number of pixel groups of the processing region, from claim 8, characterized in that to calculate the dissimilarity for each candidate of the reference pixels 10 The video decoding method according to any one of the above. The reference pixel determination step, the sum is minimized with increasing function value for the distance between the reference pixel for smoothing the sum of the dissimilarity, the variation of the reference picture element in the processing region for each reference pixel as, for each pixel or a predetermined number of pixel groups of said processing region, the image of any one of claims 8 11, characterized by determining the reference pixels from the candidates of the reference pixel Decryption method. The reference pixel determining step determines a plurality of reference pixels from the reference pixel candidates for each pixel in the processing region or a predetermined number of pixel groups based on the degree of difference,
The predictive image generation step, according to the prediction direction, the video decoding method according to any one of claims 8 12, wherein generating a predicted image using a plurality of reference pixels. The predicted image generation step includes a weighting factor setting step of setting a weighting factor according to the degree of difference for each of the plurality of reference pixels determined for each pixel in the processing region or for a predetermined number of pixel groups. ,
The video decoding method according to claim 13 , wherein a predicted image is generated using a weighted average value or a weighted median value of the plurality of reference pixels using the weighting coefficient according to the prediction direction. A video encoding device that performs predictive encoding using intra-frame prediction on a processing region obtained by dividing each frame constituting the video into a predetermined size while using a depth map corresponding to the video. There,
An intra-frame prediction direction setting means for setting a prediction direction in the intra-frame prediction;
For each pixel in the processing area or for a predetermined number of pixels, a pixel that exists in a direction opposite to the set prediction direction and is already encoded within a predetermined distance range from the processing area is a reference pixel candidate. Reference pixel candidate setting means for setting as:
Using the depth map corresponding to the processing region and the depth map corresponding to the reference pixel candidate, a difference degree for each reference pixel candidate is calculated for each pixel in the processing region or a predetermined number of pixel groups. Reference pixel dissimilarity calculating means
Reference pixel determining means for determining a reference pixel from among the reference pixel candidates for each pixel in the processing region or a predetermined number of pixel groups based on the degree of difference;
A video encoding device, comprising: predicted image generation means for generating a predicted image for the processing region from the reference pixel according to the set prediction direction. The reference pixel candidate setting means, in addition to the reference pixel candidate, a direction orthogonal to the set prediction direction for the reference pixel candidate for each pixel in the processing region or a predetermined number of pixel groups. 16. The video encoding apparatus according to claim 15, wherein an already encoded pixel that is present in a predetermined distance range from the reference pixel candidate is also set as the reference pixel candidate. In addition to the reference pixel candidates, the reference pixel candidate setting means has a predetermined angle range centered on a direction opposite to the set prediction direction for each pixel in the processing region or a predetermined number of pixel groups. 16. The video encoding apparatus according to claim 15, wherein an already encoded pixel that exists and is within a predetermined distance range from the processing region is also set as the reference pixel candidate. A video decoding device that decodes a video signal using intra-frame prediction for a processing region obtained by dividing each frame constituting the video into a predetermined size while using a depth map corresponding to the video. There,
An intra-frame prediction direction setting means for setting a prediction direction in the intra-frame prediction;
For each pixel or a predetermined number of pixel groups of said processing region, present in the prediction direction opposite the direction that the setting, as already candidates for reference pixels decoded pixels within a predetermined distance range from the processing region Reference pixel candidate setting means for setting;
Using the depth map corresponding to the processing region and the depth map corresponding to the reference pixel candidate, a difference degree for each reference pixel candidate is calculated for each pixel in the processing region or a predetermined number of pixel groups. Reference pixel dissimilarity calculating means
Reference pixel determining means for determining a reference pixel from among the reference pixel candidates for each pixel in the processing region or a predetermined number of pixel groups based on the degree of difference;
A video decoding apparatus comprising: a predicted image generating unit configured to generate a predicted image for the processing region from the reference pixel according to the set prediction direction. The reference pixel candidate setting means, in addition to the reference pixel candidate, a direction orthogonal to the set prediction direction for the reference pixel candidate for each pixel in the processing region or a predetermined number of pixel groups. 19. The video decoding apparatus according to claim 18, wherein already decoded pixels that are present in a predetermined distance range from the reference pixel candidates are also set as the reference pixel candidates. In addition to the reference pixel candidates, the reference pixel candidate setting means has a predetermined angle range centered on a direction opposite to the set prediction direction for each pixel in the processing region or a predetermined number of pixel groups. 19. The video decoding apparatus according to claim 18, wherein an already decoded pixel that exists and is within a predetermined distance range from the processing region is also set as the reference pixel candidate. A video encoding program for causing a computer to execute the video encoding method according to any one of claims 1 to 7 . A video decoding program for causing a computer to execute the video decoding method according to any one of claims 8 to 14 .

Images