JP2012120108A - Interpolation image generating apparatus and program, and moving image decoding device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress quality degradation of generated images even when a processing amount is reduced, in the processing of generating an interpolation object image for interpolation between a first reference image and a second reference image on the time sequence of moving images.SOLUTION: This invention is related to an interpolation image generating apparatus for generating the interpolation object image for the interpolation between the first reference image and the second reference image. The interpolation image generating apparatus includes means for generating one or more candidate vectors to be candidates of a motion vector between the first reference image and the second reference image for each processing unit area of an interpolation image and means for generating a reference vector scaled to a vector to each reference image for each of the generated motion vectors, adjusts the generated reference vector to common pixel position accuracy in common to the respective images constituting the moving images, evaluates and selects the origin candidate vector using the adjusted vector, and generates the image of the processing unit area using the adjusted vector generated from the selected candidate vector.

Description

  The present invention relates to an interpolated image generation device and program, and a moving image decoding device and program, and can be applied to, for example, those using a distributed video coding method (hereinafter referred to as a DVC method). is there.

  Conventionally, there is a DVC method as a moving image encoding method. In the DVC method, first, a sequence of video frames in a moving image is divided into a key frame (Key frame) and a Wyner-Ziv (hereinafter referred to as “WZ”) frame. In the DVC system, the key frame is, for example, conventional MPEG, H.264, or the like. Intraframe coding in H.264 / AVC or the like is performed.

  In the DVC method, the WZ frame is encoded by applying an error correction code technique such as the Slepian-Wolf theorem or the Wyner-Ziv theorem. Specifically, on the side of the encoding device that encodes the WZ frame, the Slepian-Wolf encoding process is performed on the original image to be encoded. On the decoding device side that decodes the encoded data, the WZ frame image between the key frames is predicted from the decoded key frames, and side information (sub-information; sub information; WZ decoding) is based on the predicted image. Side information) is generated. Then, on the decoding device side, Slepian-Wolf decoding is performed based on the encoded data and the side information to obtain a decoded image of the WZ frame.

  Conventionally, as a technique for estimating a WZ frame image between key frames on the decoding device side, there is a technique described in Non-Patent Document 1. In Non-Patent Document 1, bi-directional motion estimation (bidirectional ME) and motion compensated interpolation (Motion Compensated interpolation) are used when generating a predicted image of a WZ frame.

  In the bi-directional motion estimation in Non-Patent Document 1, the motion vector of an image (hereinafter, also simply referred to as “vector”) from the WZ frame to be predicted to the previous and subsequent frames in time series is symmetric. Assuming this, candidate motion vectors between key frames are selected based on criteria such as MAD (Mean Absolute Difference) between referenced key frames. In the technique described in Non-Patent Document 1, a predicted image of the WZ frame is generated using the obtained motion vector.

  Further, in the technique described in Non-Patent Document 1, when generating a predicted image of a WZ frame, a pixel that is more detailed than the pixel position accuracy in a series of frames (a WZ frame and a key frame to be predicted) constituting a moving image. Motion estimation (half-pixel accuracy motion estimation) with position accuracy (hereinafter also referred to as “sub-pixel accuracy”) is performed.

X. Artigas, J. et al. Ascenso, M.M. Dalay, S .; KIomp, D.M. Kubasov and M.K. Ouaret, “The Discover Codec: Architecture, Techniques and Evaluation,” in Proc. of Picture Coding Symposium (PCS'07), VoI. 6, pp. 14496-14410, Lisbon, Portugal, Nov. 2007.

  In motion compensated prediction image generation described in Non-Patent Document 1, reference is made to a motion vector between key frames, even when searching and selecting a vector with pixel position accuracy in a key frame as a vector between key frames. When scaling to a time interval between a key frame and a WZ frame, it is necessary to refer to the key frame with sub-pixel accuracy. For this reason, the technique described in Non-Patent Document 1 requires data processing with sub-pixel accuracy, which may increase the processing load and memory usage.

  Further, in the prior art, only a vector coarser than the pixel position accuracy multiplied by the time interval is used as an inter-key frame vector so that the reference is made with pixel position accuracy (so that motion estimation processing with sub-pixel accuracy does not occur). There is a method of searching and selecting (for example, at intervals of two pixels). However, in this case, there is a case where an effective vector cannot be selected, and there is a problem that the image quality deteriorates.

  In view of the above-described problems, an interpolated image (for example, a WZ frame) that interpolates between the first reference image and the second reference image (for example, between key frames) on the time series of moving images. ), An interpolated image generating device and program, and a moving image decoding device and program that can suppress the quality degradation of the generated image even if the processing amount is reduced are desired.

  According to a first aspect of the present invention, there is provided an interpolated image generating apparatus for generating an interpolated image for interpolating between a first reference image and a second reference image on a time series of moving images. (1) Processing of the interpolated image For each unit region, motion vector candidate generation means for generating one or more candidate vectors as motion vector candidates between the first reference image and the second reference image that pass through the processing unit region; 2) For each candidate vector generated by the motion vector candidate generation means, a first reference vector scaled from the interpolated image to the first reference image, and the interpolated image to the second reference image Reference vector generation means for generating a second reference vector scaled into a vector to (3), and (3) a first reference vector and a second reference vector generated by the reference vector. (4) Reference vector accuracy adjusting means for adjusting the common pixel position accuracy common to each image constituting the moving image to generate a first accuracy adjusted reference vector and a second accuracy adjusted reference vector; A vector evaluation unit that performs evaluation using the first accuracy-adjusted reference vector and the second accuracy-adjusted reference vector generated by the reference vector accuracy adjustment unit based on the candidate vector for each candidate vector; 5) vector selection means for selecting any candidate vector based on the evaluation result of the vector evaluation means; (6) a first accuracy-adjusted reference vector based on the candidate vector selected by the vector selection means; Interpolated image generating means for generating an image for each processing unit area of the interpolated image using the accuracy-adjusted reference vector of 2. To.

  According to a second aspect of the present invention, there is provided a key frame decoding means for decoding encoded data of a key frame separated from a frame sequence of a moving image, and a prediction for a non-key frame separated from the frame sequence from information relating to the key frame. Predicted image generating means for generating an image using an interpolation image generating device for generating an interpolated image for interpolating between a first reference image and a second reference image on a time series of moving images, and the predicted image In the moving picture decoding apparatus having the Slepian-Wolf decoding means for decoding the encoded data of the non-key frame using the prediction image generated by the generation means, the first aspect of the present invention is used as the interpolation image generation apparatus. It is used.

  A computer program installed in an interpolation image generation apparatus for generating an interpolation image for interpolating between a first reference image and a second reference image on a time series of moving images. (1) For each processing unit region of the interpolated image, one or more candidate vectors serving as motion vector candidates between the first reference image and the second reference image passing through the processing unit region (2) a first reference vector scaled to a vector from the interpolated image to the first reference image for each of the candidate vectors generated by the motion vector candidate generating unit; A reference vector generating means for generating a second reference vector scaled into a vector from the interpolated image to the second reference image; and (3) a first generated by the reference vector. The reference vector and the second reference vector are adjusted to the common pixel position accuracy common to the images constituting the moving image, and the first accuracy-adjusted reference vector and the second accuracy-adjusted reference vector are generated. A reference vector accuracy adjusting means, and (4) for each candidate vector, the first accuracy adjusted reference vector and the second accuracy adjusted reference vector generated by the reference vector accuracy adjusting means based on the candidate vector are used. Vector evaluation means for performing evaluation, (5) vector selection means for selecting one of the candidate vectors based on the evaluation result of the vector evaluation means, and (6) a first number based on the candidate vector selected by the vector selection means. An image for each processing unit area of the interpolated image is generated using the first accuracy-adjusted reference vector and the second accuracy-adjusted reference vector. Characterized in that to function as an interpolation image generating means for.

  A moving picture decoding program according to a fourth aspect of the present invention includes: (1) key frame decoding means for decoding encoded data of a key frame separated from a frame sequence of moving pictures; and (2) information relating to the key frame. To generate an interpolated image for interpolating between the first reference image and the second reference image, on the time series of the moving image, the predicted image for the non-key frame separated from the frame sequence (3) using the predicted image generated by the predicted image generating means, and functioning as a Slepian-Wolf decoding means for decoding the encoded data of the non-key frame, (4) The program part functioning as the interpolation image generation device is (4-1) for each processing unit area of the interpolation image, the processing unit area Motion vector candidate generation means for generating one or more candidate vectors as motion vector candidates between the first reference image and the second reference image that pass through; and (4-2) the motion vector candidate generation means. For each of the candidate vectors generated by the first reference vector scaled to a vector from the interpolated image to the first reference image, and a second scaled to a vector from the interpolated image to the second reference image. (4-3) Concerning the first reference vector and the second reference vector generated by the reference vector, the first reference vector is configured as the moving image. A reference vector that generates a first accuracy-adjusted reference vector and a second accuracy-adjusted reference vector by adjusting the common pixel position accuracy common to the images to be processed. And (4-4) for each candidate vector, the first accuracy-adjusted reference vector and the second accuracy-adjusted reference vector generated by the reference vector accuracy adjusting unit based on the candidate vector are used. (4-5) a vector selection unit for selecting any candidate vector based on the evaluation result of the vector evaluation unit, and (4-6) the vector selection unit selected. Interpolation image generation means for generating an image for each processing unit region of the interpolation image using the first accuracy-adjusted reference vector and the second accuracy-adjusted reference vector based on the candidate vector, To do.

  According to the present invention, in the process of generating an interpolated image that interpolates between the first reference image and the second reference image on the time series of moving images, even if the processing amount is reduced, the generated image Can suppress quality degradation

It is the block diagram shown about the functional structure of the interpolation image generation apparatus which concerns on 1st Embodiment. It is a block diagram which shows the whole structure of the moving image delivery system which concerns on 1st Embodiment. It is the flowchart shown about operation | movement of the interpolation image generation apparatus which concerns on 1st Embodiment. It is explanatory drawing shown about a mode that the precision of a motion vector is adjusted in the interpolation image generation apparatus which concerns on 1st Embodiment. It is the block diagram shown about the functional structure of the interpolation image generation apparatus which concerns on 2nd Embodiment. It is the flowchart shown about operation | movement of the interpolation image generation apparatus which concerns on 2nd Embodiment.

(A) First Embodiment Hereinafter, a first embodiment of an interpolated image generation apparatus and program, and a moving image decoding apparatus and program according to the present invention will be described in detail with reference to the drawings.

(A-1) Configuration of the First Embodiment FIG. 2 is a block diagram showing the overall configuration of the moving image distribution system 1 of the first embodiment.

  In the moving image distribution system 1, a moving image encoding device 2 and a moving image decoding device 3 are arranged.

  First, the moving image encoding device 2 will be described.

  The moving image encoding device 2 includes a distributed encoder 21 and a key frame encoder 22. Note that the moving picture encoding apparatus 2 can apply the same configuration as the moving picture encoding apparatus that generates encoded data of moving picture data by the existing DVC method, and therefore, detailed description is omitted here. To do.

  In the moving image encoding device 2, the frame of the input moving image is classified into a WZ frame or a key frame. The key frame is encoded by a predetermined intra-frame encoding by the key frame encoder 22 and transmitted to the decoding side (moving image decoding apparatus 3).

  On the other hand, the WZ frame is encoded by the distributed encoder 21, and the encoded data is transmitted to the decoding side (moving image decoding apparatus 3).

  Here, for example, the distributed encoder 21 converts a WZ frame to be encoded into a transform coefficient region (frequency region) (DCT or the like), quantizes each component in the transform coefficient region, and converts the quantized value into a bit string. For example, the information of each bit is subjected to Slepian-Wolf coding for each information (bit plane) collected for one frame, and among the results, a predetermined amount of parity bits or the like is used as encoded data. It shall be transmitted to the decoding side.

  Next, the configuration of the video decoding device 3 will be described.

  The moving image decoding device 3 is a device that decodes moving image data (data encoded using the DVC method) received from the encoding side (the moving image encoding device 2).

  The moving image decoding apparatus 3 includes a distributed decoder 31, a key frame decoder 32, a frame buffer 33, a side information generation unit 34, and an interpolated image generation apparatus 35. In the moving picture decoding apparatus 3, the same configuration as that of the moving picture decoding apparatus that decodes the encoded data encoded by the existing DVC method can be applied to the configuration other than the interpolation image generating apparatus 35 described later. Detailed explanation is omitted because it is possible.

  The moving image decoding device 3 may be constructed by connecting various circuits in hardware, and a general-purpose device having a CPU, a ROM, a RAM, and the like executes a moving image decoding program, thereby moving image decoding It may be constructed so as to realize a function as a device. Regardless of which construction method is applied, the functional configuration of the video decoding device 3 can be represented in FIG.

  The distributed decoder 31 uses the side information (data based on the predicted image of the WZ frame) supplied from the side information generation unit 34 for the encoded data of the WZ frame among the encoded data input to the video decoding device 3. And performs decoding (Slepian-Wolf decoding process). For example, a process of decoding encoded data encoded by the existing DVC method can be applied to the decoding process by the distributed decoder 31.

  Specifically, when the distributed decoder 31 receives the encoded data (parity bits or the like) for each bit plane, the distributed decoder 31 performs a Slepian-Wolf decoding process from the encoded data and the side information generated by the side information generating unit 34. To generate a decoded image of the WZ frame.

  The key frame decoder 32 decodes the encoded data of the key frame among the moving image data input to the moving image decoding device 3 to obtain a decoded image. In addition, the distributed decoder 31 may give the data of the decoded image of the WZ frame to the frame buffer 33 for storage.

  The frame buffer 33 stores the decoded image of the key frame decoded by the key frame decoder 32 and outputs the decoded image of the stored key frame.

  The side information generation unit 34 generates a predicted image of the WZ frame based on the decoded image stored in the frame buffer 33, and supplies data based on the predicted image to the distributed decoder 31 as side information. The side information generation unit 34 uses the interpolation image generation device 35 to generate a predicted image of the WZ frame.

  As described above, the interpolated image generation device 35 is a device used in the side information generation unit 34 to generate a predicted image of the WZ frame. When two reference images in the moving image are input, the time series of the moving image is obtained. An interpolated image for interpolating between the two reference images is generated above. In the interpolation image generation device 35, the interpolation target image is a WZ frame, and for example, a decoded image of key frames before and after the WZ frame is used as a reference image to be referred to when generating the interpolation target image. The details of the interpolation image generation device 35 will be described later.

  The side information generation unit 34 may supply the predicted image (interpolation image) of the WZ frame generated by the interpolation image generation device 35 to the distributed decoder 31 as it is, or a format required by the distributed decoder 31 (for example, DCT converted and processed into a bit plane) may be supplied. Note that the process of converting the predicted image (interpolated image) of the WZ frame into a format required for the decoding process in the distributed decoder 31 may be performed on the distributed decoder 31 side or on the side information generating unit 34 side. The way of sharing is not limited.

  The output format of the decoded image by the moving image decoding device 3 is not limited. For example, moving image data based on the decoded image (for example, data in MPEG format) may be output, or video in a predetermined format You may output as a signal (for example, video signal which can be supplied to a display).

  Next, details of the interpolated image generation device 35 will be described.

  FIG. 1 is a block diagram showing a functional configuration of the interpolated image generation device 35.

  The interpolated image generation device 35 includes a first reference image storage unit 351, a second reference image storage unit 352, a motion vector candidate generation unit 353, a first reference vector accuracy adjustment unit 354, and a second reference vector accuracy adjustment unit. 355, a motion vector selection unit 356, and an interpolation image generation unit 357.

  The interpolated image generation device 35 may be constructed by connecting various circuits in hardware, and is a general-purpose device (including other components in the video decoding device 3) having a CPU, a ROM, a RAM, and the like. However, it may be constructed so as to realize a function as an interpolated image generation device by executing a predicted image generation program. Regardless of which construction method is applied, the functional configuration of the interpolation image generation device 35 can be represented in FIG.

  The first reference image storage unit 351 and the second reference image storage unit 352 are storage means for storing the decoded image data of the key frame referred to when the interpolation image generation device 35 generates the predicted image of the WZ frame ( Memory).

  Here, the first reference image storage unit 351 stores the decoded image of the key frame that is ahead of the WZ frame in the time series (previous to the WZ frame) with respect to the target WZ frame for which the interpolation image generation device 35 generates the interpolation image. Data shall be stored. Then, the second reference image storage unit 352 stores the decoded image data of the key frame that is later in the time series than the WZ frame for which the interpolation image generation device 35 generates the interpolation image (future with respect to the WZ frame). Shall be stored.

  Here, the first reference image storage unit 351 stores the image data of the key frame immediately before the WZ frame, and the second reference image storage unit 352 stores the image data of the key frame immediately after the WZ frame. Although described as being stored, the image used as the reference image is not limited to this. In FIG. 1, a first reference image storage unit 351 and a second reference image storage unit 352 are provided in the interpolated image generation device 35 to hold key frame image data. The storage means may be omitted and the data in the frame buffer 33 may be simply referred to.

  Hereinafter, a key frame stored in the first reference image storage unit 351 is referred to as a “first reference image”, and a key frame stored in the second reference image storage unit 352 is referred to as a “second reference image”. Shall be called.

  The motion vector candidate generation unit 353 is a motion vector candidate between the first and second reference images that passes through the region of the processing block in units of processing blocks (for example, M × N rectangular blocks) of the interpolation target image. Are generated with the pixel position accuracy of the reference image.

  Specifically, the motion vector candidate generation unit 353 selects candidates for vectors from the first reference image to the second reference image (those that pass through the region of the processing block) for each processing block of the interpolation target image. It is generated and supplied to the first reference vector accuracy adjustment unit 354 and the second reference vector accuracy adjustment unit 355.

  The first reference vector accuracy adjustment unit 354 generates a vector obtained by scaling the vector (vector between reference images) supplied from the motion vector candidate generation unit 353 to a vector from the interpolation target image to the first reference image. Further, the vector is adjusted to the pixel position accuracy of the reference image and supplied to the motion vector selection unit 356.

  The second reference image storage unit 352 generates a vector obtained by scaling the vector supplied from the motion vector candidate generation unit 353 (vector between reference images) to a vector from the interpolation target image to the second reference image, Further, the vector is adjusted to the pixel position accuracy of the reference image and supplied to the motion vector selection unit 356.

  Details of the processes of the first reference vector accuracy adjustment unit 354 and the second reference vector accuracy adjustment unit 355 will be described later.

  The motion vector selection unit 356 evaluates each of the candidate vectors generated by the motion vector candidate generation unit 353 using a predetermined evaluation criterion, and selects an appropriate vector for generating the interpolation target image for each processing block. . Specifically, the motion vector selection unit 356 includes, for each processing block of the interpolation target image, an image of the first reference image area referred to by the vector supplied from the first reference vector accuracy adjustment unit 354, and In addition, a process of evaluating the image of the second reference image area referred to by the vector supplied from the second reference vector accuracy adjusting unit 355 according to a predetermined evaluation criterion may be performed.

  The interpolation image generation unit 357 generates an interpolation target image (predicted image of the WZ frame) using the vector selected by the motion vector selection unit 356 and the first and second reference images. Specifically, the interpolation image generation unit 357 uses the images of the first and second reference image areas specified by the vector selected by the motion vector selection unit 356 for each processing block of the interpolation target image. An image is generated.

(A-2) Operation of the First Embodiment Next, the operation of the interpolated image generation device 35 in the video decoding device 3 of the first embodiment having the above configuration will be described. In the moving image decoding device 3, operations other than the interpolated image generating device 35 can be applied with the same processing as the existing moving image decoding processing, and thus detailed description thereof is omitted.

  FIG. 3 is a flowchart showing the operation of the interpolated image generating device 35.

  In FIG. 3, reference images (first and second reference images) used for generating an arbitrary interpolation target image (predicted image of a WZ frame) in advance for the interpolated image generating device 35 are the side information generating unit 34. It is assumed that the input is set by. Then, in the interpolation image generation device 35, the processing of the flowchart shown in FIG. 3 is performed for each processing block of the interpolation target image.

  First, the interpolation image generation device 35 initializes the motion vector V for the interpolation target image (for example, zero vector (V = 0)) (S101).

  Next, based on a plurality of motion vector search strategy processes (assuming a predetermined method), a motion vector between reference images (from the second to the first reference image) is generated by the motion vector candidate generation unit 353. Motion vector) candidates are sequentially generated and supplied to the first reference vector accuracy adjustment unit 354 and the second reference vector accuracy adjustment unit 355 (S102). At this time, the motion vector candidate generation unit 353 does not provide a restriction that only a vector coarser than the pixel position accuracy is used as a vector between reference images. For the generation of motion vectors by the motion vector candidate generation unit 353, known motion vector search processing of various moving images can be applied, and therefore the description of the contents of the individual search processing is omitted.

  In the following, an arbitrary candidate of a vector (motion vector from the second to the first reference image) generated by the motion vector candidate generation unit 353 for an arbitrary processing block of the interpolation target image is represented as a vector U. Shall.

  Then, the first reference vector accuracy adjustment unit 354 and the second reference vector accuracy adjustment unit 355 generate vectors that are scaled into vectors from the interpolation target image to the respective reference images for the vector U. The vector is adjusted to the image position accuracy of the reference image (S103).

Hereinafter, a vector generated by the first reference image storage unit 351 based on the vector U and precision-adjusted is represented as a vector V ₁ . The second reference vector accuracy adjustment unit 355 is assumed to represent a vector obtained by precision adjustment and generated based on the vector U and the vector V _2. In step S103, the first reference vector accuracy adjustment unit 354 and the second reference vector are set so that the vector obtained by combining the adjusted vectors V ₁ and V ₂ becomes the vector U (U = V ₁ −V ₂ ). It is desirable that the processing is performed by the accuracy adjustment unit 355.

For example, when the interpolation target image is located in the middle of the reference image, the vectors U / 2 and -U / 2 obtained by scaling the inter-reference image vector by 1/2 are referred to the first and second reference images, respectively. It becomes a vector, and vectors V ₁ and V ₂ obtained by rounding the value of each component of the vector to the pixel position accuracy can be obtained as in the following expressions (1) and (2). However, in the following formulas (1) and (2), the function round is assumed to be a function for rounding the precision of each component of the vector, and for example, round-off, round-up, round-off, etc. satisfy U = V ₁ −V ₂ Various functions are available. In addition, “round down”, “round up”, and the like in the function “round” are positions that are gridded to the pixel position accuracy in the reference image when referring to the reference image from the interpolation target image using the vectors V ₁ and V _2. It is necessary to refer to (pixel) and avoid sub-pixel accuracy.

V ₁ = round (U / 2) (1)
V ₂ = round (−U / 2) (2)
FIG. 4 is an explanatory diagram showing an example in which the vectors V ₁ and V ₂ are adjusted in the first reference vector accuracy adjustment unit 354 and the second reference vector accuracy adjustment unit 355.

In FIG. 4, a vector U ₁ is a vector before correction of the vector V ₁ , and a vector U ₂ is a vector before correction of the vector V ₂ . As shown in FIG. 4, the vectors V ₁ and V ₂ are not necessarily symmetric vectors as a result of the adjustment, but the position of the reference image referenced thereby does not require subpixel accuracy.

Next, the motion vector selection unit 356 evaluates the vector U (vectors V ₁ and V ₂ ) between the reference images according to a predetermined evaluation standard (S104).

For example, SAD (Sum of Absolute Difference) represented by the following formula (3), an evaluation method in which a penalty term related to the magnitude of a vector is added to the image dissimilarity criterion, such as SAD, or the like is used. Can do. However, in (3), P is the pixel position in the interpolated image, sigma _p represents the sum concerning the processing block. Further, l ₁ and l ₂ represent pixel values of the first and second reference images, respectively.

SAD = Σ _p | l ₁ (P + V ₁ ) −l ₂ (P + V ₂ ) | (3)
That is, when the vector U is evaluated using the SAD expressed by the above equation (3) in the motion vector selection unit 356, the smaller the evaluation value, the better the evaluation (upper rank).

Next, the motion vector selection unit 356 determines whether or not the evaluation value for the most recently evaluated vector U is the minimum value among the previous candidates (S105), and determines that the evaluation value is the minimum value. In the case where it is detected, the vector V is updated to the latest evaluated vector U (V = U) (S108). In step S108, when the vector V is updated to the latest evaluated vector U, the motion vector selection unit 356 may hold the vectors V ₁ and V ₂ based on the vector U.

  Next, in the interpolated image generation device 35, the motion vector candidate generation unit 353 confirms whether or not there is a vector U that has not yet been evaluated by the motion vector selection unit 356 (S106), and there is an unevaluated vector U. In this case, the process returns to the above-described step S102, and the same processing is performed for the unevaluated vector U.

On the other hand, when it is determined in step S106 that there is no unevaluated vector U (when evaluation values are obtained for all vectors U), the motion vector selection unit 356 determines the final vector V ( A vector U) having the smallest evaluation value among candidates and reference vectors V ₁ and V ₂ corresponding to the vector U) are obtained, and a selection result is obtained.

Next, the interpolation image generation unit 357 uses the vectors (vectors V ₁ and V ₂ ) selected by the motion vector selection unit 356 to generate the first and second reference images for each pixel P of the interpolation target image. The pixel value is determined with reference to generate an image of the processing block in the interpolation target image (S107). The method for obtaining the pixel value of the pixel P is not limited. For example, the average pixel value of the corresponding pixel in the first and second reference images may be used, or a weighted average may be used. .

  The interpolation image generation device 35 performs the above-described processing of steps S101 to S108 for each processing block of the interpolation target image to generate and output the entire interpolation target image. Then, the side information generation unit 34 generates side information using the image generated by the interpolation image generation device 35 as a predicted image of the WZ frame.

(A-3) Effects of First Embodiment According to the first embodiment, the following effects can be achieved.

  In the interpolated image generation device 35, the reference image area referred to when the motion vector selection unit 356 selects the inter-reference image vector (vector U) and the predicted image is generated by the interpolation image generation unit 357 is referred to. The pixel position accuracy of the image is adjusted. As a result, the interpolated image generation device 35 does not require subpixel precision image generation processing at the time of evaluation for selecting a reference inter-image vector and at the time of generating a predicted image, as in the prior art. Resources such as a processing amount and a memory amount necessary for processing can be reduced.

  Furthermore, the interpolated image generation device 35 is necessary for the interpolated image generation process as described above without reducing the vector search options by reducing the accuracy of the vector as the inter-reference image vector as in the prior art. Therefore, the estimation accuracy of the motion vector is improved and the image quality of the interpolated image can be improved.

(B) Second Embodiment Hereinafter, a second embodiment of an interpolated image generation apparatus and program, and a moving image decoding apparatus and program according to the present invention will be described in detail with reference to the drawings.

(B-1) Configuration of the Second Embodiment The overall configuration of the moving image distribution system 1A of the second embodiment can also be described using FIG. 2 as in the first embodiment. Hereinafter, the difference between the second embodiment and the first embodiment will be described.

  The moving image distribution system 1A is different from the first embodiment in that the moving image decoding device 3 is replaced with the moving image decoding device 3A.

  Further, the moving image decoding device 3A is different from the first embodiment in that the interpolation image generation device 35 is replaced with the interpolation image generation device 35A.

  Next, the internal configuration of the interpolation image generation device 35A will be described.

  FIG. 5 is a block diagram illustrating a functional configuration of the interpolated image generation device 35A according to the second embodiment.

  In the interpolation image generation device 35A, a first subpixel generation unit 358 and a second subpixel generation unit 359 are added, and further, the interpolation image generation unit 357 is replaced with the interpolation image generation unit 357A. This is different from the first embodiment.

In the interpolated image generation device 35A, when the vector U is evaluated by the motion vector selection unit 356, the precision-adjusted vectors V ₁ and V ₂ are used as in the first embodiment. On the other hand, a vector before accuracy adjustment of the vectors V ₁ and V ₂ is used for generation of a prediction image by the interpolation image generation unit 357A.

In the second embodiment as well, the vectors before the vectors V ₁ and V ₂ are adjusted in precision are represented as vectors U ₁ and U ₂ , respectively.

Then, when the predicted image is generated using the vectors U ₁ and U ₂ , the interpolated image generation unit 357A requires the _first subpixel when an image representing the reference image with subpixel accuracy is required. The generation unit 358 or the second subpixel generation unit 359 is requested to generate an image with subpixel accuracy for a necessary region. The processing of the interpolated image generation unit 357A will be described in detail in the operation description to be described later.

  The first subpixel generation unit 358 generates a subpixel-accurate image of the requested area for the first reference image in response to a request from the interpolation image generation unit 357A. The second subpixel generation unit 359 generates a subpixel-accurate image of the requested area for the second reference image in response to a request from the interpolation image generation unit 357A.

  In the first subpixel generation unit 358 and the second subpixel generation unit 359, various existing methods can be used as processing for generating an image with subpixel accuracy from the reference image. These interpolation filters can be used. For example, when the key frame decoder 32 includes an interpolation filter that can generate an image with sub-pixel accuracy of a key frame, it may be used.

(B-2) Operation | movement of 2nd Embodiment Next, operation | movement of the interpolation image generation apparatus 35A in the moving image delivery system 1A of 2nd Embodiment which has the above structures is demonstrated.

  FIG. 6 is a flowchart showing the operation of the interpolated image generating device 35A.

  Since the operations of steps S201 to S206 and S208 are substantially the same as the operations of steps S101 to S106 and S108 described above in the first embodiment, detailed description thereof will be omitted.

However, in step S203 (corresponding to step S103 described above in the first embodiment), a vector obtained by scaling the vector U to a vector from the interpolation target image to each reference image is generated. The vectors are represented as vectors U ₁ and U ₂ as described above. The vectors U ₁ and U _{2 that} have been adjusted in precision become vectors V ₁ and V ₂ .

When the interpolation target image is located in the middle of the reference image, the vectors U ₁ and U ₂ are obtained by scaling the motion vector U between the reference images to ½ times, and the vectors U ₁ , U ₂ , V ₁ and V ₂ can be expressed by the following equations (4) to (7), respectively. However, since the function round in the following formulas (6) and (7) is the same as the above formulas (1) and (2), description thereof is omitted.

U ₁ = U / 2 (4)
U ₂ = −U / 2 (5)
V ₁ = round (U ₁ ) (6)
V ₂ = round (U ₂ ) (7)
When the motion vector selection unit 356 selects the vector V (selects the vector U with the best evaluation value) by the processing of steps S201 to S206 and S208, the interpolation image generation unit 357A first selects the vector V. It is determined whether the vectors U ₁ and U ₂ corresponding to V are referenced with sub-pixel accuracy of the reference image (S209).

If it is determined in step S209 that the vectors U ₁ and U ₂ refer to the reference image with sub-pixel accuracy, the interpolated image generation unit 357A includes the first sub-pixel generation unit 358 and / or The second subpixel generation unit 359 is requested to obtain an image with subpixel accuracy for the corresponding region (region referred to by the vectors U ₁ and U ₂ from the processing block) (S210). The sub-pixel accuracy image generated by the first sub-pixel generation unit 358 and / or the second sub-pixel generation unit 359 may be, for example, a half-pixel accuracy relative to the pixel position accuracy of the reference image. However, it is good also as detailed accuracy. The accuracy of image generation by the first subpixel generation unit 358 and / or the second subpixel generation unit 359 may not be fixed accuracy. For example, the level of detail for generating an image may be determined from the processing block according to the accuracy of the image referenced by the vectors U ₁ and U ₂ .

  Then, the interpolation image generation unit 357 generates an image of the processing block in the interpolation target image (S207).

If it is determined in step S209 that the vectors U ₁ and U ₂ refer to the reference image with sub-pixel accuracy, the interpolated image generation unit 357 determines that the vectors U ₁ and U ₂ The image of the processing block in the interpolation target image is generated using the sub-pixel precision reference image generated in step S210.

On the other hand, when it is determined in step S209 described above that the vectors U ₁ and U ₂ do not refer to the sub-pixels of the reference image, the interpolated image generation unit 357 performs the same process as in step S107 described above. Using the vectors (vectors V ₁ and V ₂ ) selected by the motion vector selection unit 356, an image of the processing block in the interpolation target image is generated.

  Then, the interpolated image generation device 35A performs the processing of steps S201 to S210 described above for each processing block of the interpolation target image to generate and output the entire predicted image of the interpolation target image. Then, the side information generation unit 34 generates side information using the image generated by the interpolation image generation device 35A.

(B-3) Effects of Second Embodiment According to the second embodiment, the following effects can be achieved.

  In the second embodiment, the evaluation for selecting a vector between reference images (evaluation of the vector U) by the motion vector selection unit 356 has the same effect as that of the first embodiment.

  In the second embodiment, the interpolation image generation process by the interpolation image generation unit 357A uses the reference image and vector with subpixel accuracy for some processing blocks. In comparison, the image quality of the interpolated image can be improved.

Furthermore, in the second embodiment, the first reference vector accuracy adjustment unit 354 and the second reference vector accuracy adjustment unit 355 generated according to the contents of the vectors (vectors U ₁ , U ₂ ) before accuracy adjustment generated by the first reference vector accuracy adjustment unit 355. Thus, it is determined whether or not the reference image and the vector of sub-pixel accuracy of the processing block are necessary, and for the processing block determined to be unnecessary, an interpolated image is generated using the reference image as it is as in the first embodiment. Yes. As a result, the interpolated image generating device 35A can reduce the processing amount while improving the image quality of the generated interpolated image to the same extent as the processing using the reference image with subpixel accuracy.

(C) Other Embodiments The present invention is not limited to the above-described embodiments, and may include modified embodiments as exemplified below.

(C-1) In each of the above embodiments, a case has been described in which the interpolated image generation apparatus of the present invention is used for predictive image generation for side information generation in video decoding employing the DVC method. The invention is not limited to the case of the DVC system, and can be used for various purposes such as generating predicted images from a plurality of images.

(C-2) In each of the above embodiments, the case where key frames are used as the first and second reference images has been described. However, not only key frames but also decoded WZ frame images may be used. good.

(C-3) In each of the above embodiments, an example of obtaining motion vectors for each processing block has been shown. However, for example, vector candidates are narrowed down in an area of 16 × 16 size, and vector candidates of 8 × 8 size are selected. The processing procedure is such that the motion vector generation unit, the reference vector accuracy adjustment unit, and the motion vector selection unit are repeatedly used while changing the size of the target region, and the interpolated image is generated using the obtained motion vector. Anyway.

(C-4) In each of the above embodiments, the encoding unit of the WZ frame in the moving image encoding device and the moving image decoding device has been described as an image for one frame. Encoding and decoding may be performed in block units (for example, M × N rectangular block units). In this case, in the video decoding device, the side information generation unit generates an image obtained by dividing the predicted image of the WZ frame generated by the interpolation image generation device into processing block units (or the image is further converted into a bit plane by a predetermined process). (Divided data) may be given to the distributed decoder.

  In each of the above embodiments, the WZ frame is converted into the transform coefficient region (frequency region) and processed. However, the pixel region is processed without being converted into the frequency region (encoding processing and decoding). Processing). For example, after quantizing the pixel value, it may be converted into a predetermined bit string code, and a bit plane may be configured and processed. For example, with respect to the pixel values of the pixels constituting the image of the WZ frame, a bit plane is generated like a bit plane in which only the most significant bits are collected, and a bit plane in which only the second bit is collected. You may make it apply the process similar to said each embodiment.

(C-5) In each of the above embodiments, the processing block unit to be encoded and decoded matches the processing block unit of the motion vector generated and selected by the interpolated image generation device, and further decoded by the moving image decoding device. At this time, for a processing block whose quality of side information is not more than a predetermined value, the side information may be regenerated only for the processing block.

  Specifically, in the distributed decoder, when decoding from the upper bit plane for each processing block of the WZ frame, it is determined that the quality of the side information related to the processing block in which the error correction has exceeded a predetermined level is lower than the predetermined level. You may make it do. For example, in the distributed decoder, the side prediction information (interpolated image) is different from the original image for the side information related to the processing block in which the cumulative number of error bits and the occurrence probability of error bits exceed a predetermined value. It is estimated to be.

  Then, the side information generation unit generates the predicted image (interpolated image) again only for the processing block whose quality is determined to be equal to or lower than the predetermined value using the interpolation image generation device. In the interpolated image generation device, when generating an initial predicted image (interpolated image) for each processing block, among the vectors evaluated by the motion vector selection unit, a plurality of vectors having higher evaluations are held in advance, When a predicted image (interpolated image) is regenerated for a processing block, a vector having a higher evaluation among unselected vectors may be sequentially used. For example, when the interpolation image generation apparatus performs the first regeneration for the processing block (that is, when the second interpolation image generation is performed), the motion vector selection unit has the second highest evaluation value from the top. A vector may be selected, and an image generation using the vector may be performed by the interpolation image generation unit.

  For one processing block, the number of times the side information generation unit (interpolated image generation device) regenerates the side information is not limited, but for example, playback is performed until the above predetermined quality is satisfied with the predetermined number of times being the upper limit. You may make it make it complete. In addition, as a result of regenerating the side information one or more times for the processing block, when none of the side information satisfies the above-mentioned predetermined quality, the one having the best quality among the plurality of side information May be used for the final decoding result.

  DESCRIPTION OF SYMBOLS 1 ... Moving image delivery system, 2 ... Moving image encoding device, 21 ... Distributed encoder, 22 ... Key frame encoder, 3 ... Moving image decoding device, 31 ... Distributed decoder, 32 ... Key frame decoder, 33 ... Frame buffer, 34 ... Side information generating unit 35... Interpolated image generating device 351... First reference image storage unit 352... Second reference image storage unit 353... Motion vector candidate generation unit 354. , 355... Second reference vector accuracy adjustment unit, 356... Motion vector selection unit, 357.

Claims (6)

In an interpolated image generating apparatus for generating an interpolated image for interpolating between a first reference image and a second reference image on a time series of moving images,
Motion vector candidates for generating one or more candidate vectors as motion vector candidates between the first reference image and the second reference image that pass through the processing unit region for each processing unit region of the interpolation image Generating means;
For each of the candidate vectors generated by the motion vector candidate generation means, a first reference vector scaled to a vector from the interpolated image to the first reference image, and from the interpolated image to the second reference image Reference vector generation means for generating a second reference vector scaled into a vector;
The first reference vector and the second reference vector generated by the reference vector are adjusted to the common pixel position accuracy common to the images constituting the moving image, and the first accuracy-adjusted reference vector and the second reference vector are adjusted. Reference vector accuracy adjusting means for generating a reference vector after accuracy adjustment of
For each candidate vector, vector evaluation means for performing evaluation using the first accuracy-adjusted reference vector and the second accuracy-adjusted reference vector generated by the reference vector accuracy adjustment means based on the candidate vector;
Vector selection means for selecting any candidate vector based on the evaluation result of the vector evaluation means;
Interpolated image generating means for generating an image for each processing unit region of the interpolated image using the first accuracy-adjusted reference vector and the second accuracy-adjusted reference vector based on the candidate vector selected by the vector selecting means An interpolated image generating apparatus characterized by comprising: The interpolation image generation means includes
A sub-pixel image generation unit that generates an image of a set region of the first reference image or the second reference image with a pixel position accuracy that is more detailed than the common pixel position accuracy;
When the first reference vector and / or the second reference vector corresponding to the candidate vector selected for the processing unit region by the vector selection unit is indicated with a pixel position accuracy that is more detailed than the common pixel position accuracy. Of the first reference image and / or the second reference image, the region referred to by the first reference vector and / or the second reference vector is generated using the sub-pixel image generation unit. The interpolated image generation apparatus according to claim 1, wherein an image is obtained and used to generate an image of the processing unit region. A key frame decoding means for decoding the encoded data of the key frame separated from the frame sequence of the moving image, and a predicted image for the non-key frame separated from the frame sequence from the information related to the key frame A predicted image generated by using an interpolated image generating device that generates an interpolated image for interpolating between the first reference image and the second reference image on the sequence, and a predicted image generated by the predicted image generating unit In the video decoding device having the Slepian-Wolf decoding means for decoding the encoded data of the non-key frame using
A moving image decoding apparatus using the interpolated image generating apparatus according to claim 1 or 2 as the interpolated image generating apparatus. The moving image decoding apparatus further includes image evaluation means for evaluating the quality of the predicted image generated by the predicted image generation means for each processing unit region,
The predicted image generation means generates a predicted image again using an interpolation image generation device for an image of a low quality processing unit region evaluated as a predetermined quality or lower by the image evaluation means,
When there is a processing unit area in which the predicted image has been regenerated by the predicted image generating means, the above-mentioned Slepian-Wolf decoding means has regenerated the encoded data corresponding to the processing unit area. Decode using the predicted image,
When the interpolated image generation device is instructed by the predicted image generation means to regenerate an image for any processing unit region,
The vector selection means reselects one of the candidate vectors related to the processing unit region,
The interpolated image generation means regenerates an image of the processing unit region using the first accuracy-adjusted reference vector and the second accuracy-adjusted reference vector based on the reselected candidate vector. The moving picture decoding apparatus according to claim 3. A computer mounted on an interpolation image generation device that generates an interpolation image for interpolating between a first reference image and a second reference image on a time series of moving images,
Motion vector candidates for generating one or more candidate vectors as motion vector candidates between the first reference image and the second reference image that pass through the processing unit region for each processing unit region of the interpolation image Generating means;
For each of the candidate vectors generated by the motion vector candidate generation means, a first reference vector scaled to a vector from the interpolated image to the first reference image, and from the interpolated image to the second reference image Reference vector generation means for generating a second reference vector scaled into a vector;
The first reference vector and the second reference vector generated by the reference vector are adjusted to the common pixel position accuracy common to the images constituting the moving image, and the first accuracy-adjusted reference vector and the second reference vector are adjusted. Reference vector accuracy adjusting means for generating a reference vector after accuracy adjustment of
For each candidate vector, vector evaluation means for performing evaluation using the first accuracy-adjusted reference vector and the second accuracy-adjusted reference vector generated by the reference vector accuracy adjustment means based on the candidate vector;
Vector selection means for selecting any candidate vector based on the evaluation result of the vector evaluation means;
Interpolated image generating means for generating an image for each processing unit region of the interpolated image using the first accuracy-adjusted reference vector and the second accuracy-adjusted reference vector based on the candidate vector selected by the vector selecting means An interpolated image generation program characterized by functioning as a program. Computer
Key frame decoding means for decoding the encoded data of the key frame separated from the frame sequence of the moving image;
An interpolated image for interpolating between the first reference image and the second reference image on the time sequence of the moving image of the predicted image for the non-key frame separated from the frame sequence from the information related to the key frame Predicted image generation means for generating using the generated interpolated image generation device;
Using the predicted image generated by the predicted image generating means, the non-keyframe encoded data is decoded and functioned as a Slepian-Wolf decoding means,
The program part that functions as the interpolation image generation device is
Motion vector candidates for generating one or more candidate vectors as motion vector candidates between the first reference image and the second reference image that pass through the processing unit region for each processing unit region of the interpolation image Generating means;
For each of the candidate vectors generated by the motion vector candidate generation means, a first reference vector scaled to a vector from the interpolated image to the first reference image, and from the interpolated image to the second reference image Reference vector generation means for generating a second reference vector scaled into a vector;
For the first reference vector and the second reference vector generated by the reference vector, the first reference vector is adjusted to a common pixel position accuracy common to each image constituting the moving image, Reference vector accuracy adjusting means for generating an accuracy adjusted reference vector and a second accuracy adjusted reference vector;
For each candidate vector, vector evaluation means for performing evaluation using the first accuracy-adjusted reference vector and the second accuracy-adjusted reference vector generated by the reference vector accuracy adjustment means based on the candidate vector;
Vector selection means for selecting any candidate vector based on the evaluation result of the vector evaluation means;
Interpolated image generating means for generating an image for each processing unit region of the interpolated image using the first accuracy-adjusted reference vector and the second accuracy-adjusted reference vector based on the candidate vector selected by the vector selecting means A moving picture decoding program characterized by comprising:

Images